libGL not working on non-NixOS (without setting up)

[joepie91]

• Package: openarena
• Error: /nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8/openarena-0.8.8/openarena.x86_64: error while loading shared libraries: libGL.so.1: cannot open shared object file: No such file or directory
• Platform: openSUSE 13.1, x86_64, XFCE
• Version: nix-env (Nix) 1.9

Installation works, running it does not.

sven@linux-etoq:~> nix-env -i openarena
installing ‘openarena-0.8.8’
these paths will be fetched (390.19 MiB download, 426.80 MiB unpacked):
  /nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8
fetching path ‘/nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8’...

*** Downloading ‘https://cache.nixos.org/nar/194liswh0vgrj02q7q7ww8igizn45bri6ip4m5wxa47s3ghj65q7.nar.xz’ to ‘/nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  390M  100  390M    0     0  7996k      0  0:00:49  0:00:49 --:--:-- 9247k

building path(s) ‘/nix/store/j0lc0hns2kna9sw1g3k46zbazxhsxha0-user-environment’
sven@linux-etoq:~> openarena 
/nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8/openarena-0.8.8/openarena.x86_64: error while loading shared libraries: libGL.so.1: cannot open shared object file: No such file or directory

[bjornfor]

Tricky... I wonder if it can be fixed binding openarena to mesa libGL. Then you'd at least have something that runs, although not necessarily optimally for your system.

On NixOS, programs find the selected/best OpenGL implementation here:

$ echo $LD_LIBRARY_PATH 
/run/opengl-driver/lib:/run/opengl-driver-32/lib

Something similar can of course be done with nixpkgs programs on non-NixOS (manually set $LD_LIBRARY_PATH).

CC @vcunat.

[vcunat]

There's a deliberate choice to be impure in this respect, as I view libGL* as a part of your GPU driver. I expect using your system one is better even for apps that don't report an error (most do link against our version from mesa).

If one is able, the safest approach on non-nixos should be to actually create /run/opengl-driver/lib (speaking of x86_64-linux) and put symlinks to your system libGL*. Alternatively one could just set $LD_LIBRARY_PATH to the location, but beware if it contains too many other libs...

[joepie91]

Hmm. Would this not be a good usecase for parameters (letting you pick between a system libGL* path or a mesa dependency)? Or can those not be specified from the command-line?

(I'm still pretty new to Nix)

[vcunat]

You can easily use $LD_LIBRARY_PATH to open any libGL version. I've only seen one person who wanted to use the bundled mesa instead of system libGL (and on NixOS it's often the same one).

[domenkozar]

couldn't we require a nixpkgs config setting to be set and wrap all libGL software with it? that way users would get a message how/where to set the setting

[vcunat]

@domenkozar: I don't understand what you mean. Note that the problems only happen on non-nixos, and different distros have libGL on different locations, I think.

[domenkozar]

I'm saying we could use nixpkgs config to set $LD_LIBRARY_PATH for non-nixos users

[joepie91]

To a non-technical user, what would the correct (current) procedure look like for installing openarena on a non-NixOS distribution? I'm technical, but I don't know much about compiler toolchains, so an ELI5-style explanation might be the best way to make sense of this.

[joepie91]

After talking about this with @ttuegel for a while, we've established that the following works:

1. Install patchelf (nix-env -i patchelf)
2. Copy the system libGL.so.1 (usually in /usr/lib64) somewhere, eg. ~/.libgl-shim/libGL.so.1.
3. patchelf --set-rpath /lib:/usr/lib:/lib64:/usr/lib64 ~/.libgl-shim/libGL.so.1 (adjust lib paths if necessary for your distribution)
4. Replicate other libGL.so.* symlinks that your system has, but in your ~/.libgl-shim, and pointing to your modified copy.
5. Run openarena as follows: LD_LIBRARY_PATH=~/.libgl-shim openarena

This approach also seems automatable, and should work for other things that use makeWrapper :)

I'm still working on something, but once I've gotten further with that, I'll probably do a proper writeup of this on my blog or something.

[bjornfor]

What does this give beyond what the simpler LD_LIBRARY_PATH=/usr/lib64 openarena provides (substitute for whatever path libGL.so.1 has on your system)? It seems you still need to provide the custom path to libGL, which cannot be automated.

[joepie91]

@bjornfor You'd only load the system libGL, and the rest of the dependencies would remain the pure Nix variants.

[bjornfor]

Good point :-)

[bjornfor]

What about LD_PRELOAD=/lib/libGL.so.1 openarena then?

[joepie91]

That wouldn't work because of libGL dependencies:

sven@linux-etoq:~/projects/nixpkgs/test> LD_PRELOAD=/usr/lib64/libGL.so.1 openarena
/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/bash: error while loading shared libraries: libglapi.so.0: cannot open shared object file: No such file or directory

It'd only look in the Nix store, since Nix uses a custom ld. By elfpatching it, we have a copy of libGL that will also look for its own dependencies in the system-wide library directories, without affecting non-libGL dependencies of the application.

[bjornfor]

I see. I'm going to stop suggesting alternatives now :-)

[rkoe]

I tried this according to the instructions of joepie91 above, since I wanted to use OpenSCAD from Nix on Debian Jessie with Intel HD graphics (Haswell). Copying/patching libGL.so.1 was not enough, and glxinfo/OpenSCAD/OpenArena failed because they couldn't find i965_dri.so. Adding i965_dri.so also wasn't enough, because i965_dri.so needed some more libraries to be loaded. After adding these libraries, it worked. :)

The contents of my ~/.nix-libs is now:

libGL.so -> libGL.so.1.2.0
libGL.so.1 -> libGL.so.1.2.0
libGL.so.1.2.0
libpciaccess.so.0
dri/i965_dri.so
# edit: removed libz.so.1

Commands used:

mkdir ~/.nix_libs
rsync -aszvi /usr/lib/x86_64-linux-gnu/libGL.* ~/.nix_libs/
patchelf --set-rpath /lib:/usr/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu ~/.nix_libs/libGL.so.1.2.0

mkdir ~/.nix_libs/dri
rsync -aszvi /usr/lib/x86_64-linux-gnu/dri/i965_dri.so ~/.nix_libs/dri/
patchelf --set-rpath /lib:/usr/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu ~/.nix_libs/dri/i965_dri.so

cp -pi /nix/store/0fp3kp6kx12wqs593xwk7l6pbq0dkrdp-libpciaccess-0.13.4/lib/libpciaccess.so.0 ~/.nix_libs/

# edit: now works without the following lines
# (for OpenArena additionally:)
# (cp -pi /nix/store/31w31mc8immhpnmxvcl4l0fvc3i5iwh0-zlib-1.2.8/lib/libz.so.1 ~/.nix_libs/)

How to find out which libraries are required in ~/.nix_libs for some program:

1. Follow joepie91's instructions:

   mkdir ~/.nix_libs
   rsync -aszvi /usr/lib/x86_64-linux-gnu/libGL.* ~/.nix_libs/
   patchelf --set-rpath /lib:/usr/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu ~/.nix_libs/libGL.so.1.2.0
   

2. Run your program (e.g. glxinfo, openscad or openarena) with LD_LIBRARY_PATH:

   LD_LIBRARY_PATH=~/.nix_libs openscad
   

3. If the program does not start or OpenGL does not work, search the error-message in the output (e.g.: libGL error: unable to load driver: i965_dri.so)

4. Run the program again with strace:

   LD_LIBRARY_PATH=~/.nix_libs strace openscad 2>&1 | less -S
   

   and search the output for the error-message and the missing library or the fatal ENOENT.

5. Make sure that this missing library can be found by copying it into ~/.nix_libs (or a subdirectory) and patchelf-ing it (see step 1).

6. Go to step 2.

By the way, glxinfo still aborts with glxinfo: : ATUSHûHìH-Ó : Error 18644416 (same for glxgears), but OpenSCAD and OpenArena work.

[vcunat]

• LD_LIBRARY_PATH=~/.nix_libs is slightly dangerous with having standard libs in there (libz in particular). It means that all nix-installed stuff will use that libz instead of the one they were compiled against. But if the list is very small, it might cause no problems (and it's rather easy to set up).
• DRI drivers are located by libGL – I'm not sure how distros typically configure libGL's search path; it might even differ.
• For inspecting (broken dynamic) linkage it seems easiest to run ldd on the executables or libraries, but it won't show _dlopen_ed libs and some programs do use that to open libGL.
• There are also some other libs taken from that path, e.g. libvdpau_*.so. (I don't remember how libva works ATM.)

[rkoe]

LD_LIBRARY_PATH=~/.nix_libs is slightly dangerous with having standard libs in there (libz in particular). It means that all nix-installed stuff will use that libz instead of the one they were compiled against.

Since I only call some OpenGL-needing-programs with LD_LIBRARY_PATH=~/.nix_libs, only those use that libz.
I would of course like to remove the standard libs from my ~/.nix_libs, but I don't know how to do this without breaking the OpenGL-needing-programs, since the libs are only searched in .nix_libs and some "wrong" nix-store-paths:

open("/nix/store/nkp9sq3jj09wjxmq4h5zyaa0spc6nsrs-SDL-1.2.15/lib/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/nix/store/ikgb8896hqcy7pz7n35v1y6qcifim9bi-libogg-1.3.2/lib/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/nix/store/w0p7bn2rzyd4q8haggr5z7pyqh9kmmgk-libvorbis-1.3.5/lib/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/user/.nix_libs/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

(But I've removed libz now, since strangely it now works without ~/.nix_libs/libz.so.1 after uninstalling and reinstalling...)

DRI drivers are located by libGL

Unfortunately, the i965_dri.so is only searched in the wrong places, so I had to copy it to .nix_libs/dri/tls/i965_dri.so:

open("/home/user/.nix_libs/dri/tls/i965_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/user/.nix_libs/dri/i965_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/dri/tls/i965_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/dri/i965_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

For inspecting (broken dynamic) linkage it seems easiest to run ldd on the executables or libraries

Yes, but neither ldd nor LD_DEBUG helped here, since they did not show the missing libraries.

[vcunat]

Unfortunately, the i965_dri.so is only searched in the wrong places, so I had to copy it to .nix_libs/dri/tls/i965_dri.so:

That part is a very good solution, I think, except symlink should be better than copy (due to your OS updates).

Yes, but neither ldd nor LD_DEBUG helped here, since they did not show the missing libraries.

Hmm, then I suppose these are also cases using dlopen, unfortunately. I know nothing better than strace for that, which is what you used.

[mcmtroffaes]

I just ran into a similar issue trying to get nix's blender running on Fedora: blender wouldn't find i965_dri.so. Here are the steps that I had to do to make this work - notably I also had to set LIBGL_DRIVERS_PATH in addition to LD_LIBRARY_PATH (hence why I'm posting this here - this hasn't been mentioned above yet):

1. Create a folder for any additionally required OpenGL drivers/libraries.

2. Run

   LIBGL_DEBUG=verbose LD_LIBRARY_PATH=/path/to/drivers LIBGL_DRIVERS_PATH=/path/to/drivers blender
   

   If this works, great! You can drop LIBGL_DEBUG=verbose in future invokations.

3. If it doesn't work due to a missing file, create a symlink to the missing file from your driver folder and go back to step 2.

LD_DEBUG=libs may help too in step 2 in case you don't get enough output from LIBGL_DEBUG=verbose - but I didn't have to use it.

[davidak]

i can reproduce it on elementary OS Freya (based on Ubuntu 14.04 LTS).

i copied the systems libGL.so and patched it, then openarena works.

# apt-get install libgl1-mesa-dev
$ cp /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0 ~/.nix_libs/
$ patchelf --set-rpath /lib:/usr/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu ~/.nix_libs/libGL.so.1.2.0
$ cd ~/.nix_libs
$ mv libGL.so.1.2.0 libGL.so
$ LD_LIBRARY_PATH=~/.nix_libs openarena

[vcunat]

It still seems rather inconvenient that one needs to do update libGL by hand (after the host OS updates).

[evils]

i can't figure out what the actually issue is
it seems it's at least partially due to what GPU is being used being an impurity
and for some reason we can't fall back on the system GL?

can someone state what the issue is?

[NovaViper]

I'm actually having the same issue on NixOS 23.05 with nixpkgs unstable; for some reason the libGL.so.* and libEGL.so.* files don't exist in the opengl lib paths at all! The ones I have are for mesa and nvidia. How do I get the generic ones?

[vcunat]

There are no universal ones, I believe. I mean, nowhere, not just on NixOS.

[NovaViper]

There are no universal ones, I believe. I mean, nowhere, not just on NixOS.

I was meaning in particular those missing libGL.so./libEGL.so. files, is there a way to get those installed onto NixOS? I got a couple of packages (gamemode in particular) screaming that they can't run because of the missing libGL.so.1 and libEGL.so.1 files

[vcunat]

This github issue is about non-NixOS. Speaking of games on NixOS, my first guess is that you're missing hardware.opengl.driSupport32Bit = true; option.

[NovaViper]

Yeah but I also saw some mention that the issue also occurs on NixOS aswell, was what caught my attention. And I actually do have all of the options for opengl enabled

  # Enable OpenGL
  hardware.opengl = {
      enable = true;
      driSupport = true;
      driSupport32Bit = true;
      extraPackages = with pkgs; [ libva-utils ];
  };

[twhitehead]

@NovaViper that is because libGL.so is a vendor-neutral dispatch library. It has no specific GPU hardware code in it. It is just another library like all the others. No special treatment required. You can read more about that here.

What does have the special code are the libGLX_<vendor>.so and libEGL_<vedor>.so backend files that the generic libGL.so loads. You will find these under your /run/opengl-driver* directories.

[Cloudef]

I did this interesting zig program that runs compiled mach-engine binary
https://gist.github.com/Cloudef/b8c056ee6aff55aac5b5266e88876e76

If distros provided this kind of mapping
https://gist.github.com/Cloudef/b8c056ee6aff55aac5b5266e88876e76#file-runtime-zig-L113
either as a binary that can be called or file somewhere (real or virtual), it could be quite distro agnostic.

[vcunat]

No, I'm quite sure it won't work as easily. The libraries like libGL that you find on the OS are not stand-alone. They link against other libraries (libc, usually libstdc++ and others). Your application typically also uses libc, libstdc++ etc. but sometimes it was built against incompatible versions of them, so when you load everything together, you're unable to choose a set of libraries satisfying both your app and libGL.

[Cloudef]

The typical .so's that games and such loads talk with C ABI which are binary compatible. C++ libs are different thing, but programs usually don't try to load those externally. Of course libc structures being passed over DLL boundaries can be problematic.

[vcunat]

We get issues even with glibc on NixOS, as it's only compatible in a single way (upgrading, not downgrading). Other C libraries are even worse, e.g. libwayland IIRC.

[tomfitzhenry]

I'm not proud of this workaround, but if:

• you want to run a NixOS VM that relies on OpenGL (e.g. virtio-gpu-gl), on a non-NixOS machine
• you'd prefer not to require special setup on the host (e.g. wrapping things with NixGL)
• you're happy to just run with the host's QEMU (which has the advantage that it can find the host's OpenGL drivers)
• you want to use virtualisation.qemu to control the QEMU flags

... then you can do something horrible and impure like:

    qemu.package = pkgs.runCommand "non-nix-qemu" {} ''
      cp -r ${pkgs.qemu_kvm} $out
      chmod +w $out/bin/qemu-kvm
      printf '#!/bin/sh\nqemu-system-x86_64 -enable-kvm "$@"' > $out/bin/qemu-kvm
    '';

This wraps pkgs.qemu_kvm, and overwrites the qemu-kvm binary to just call qemu-system-x86_64, which in PATH on a non-NixOS machine is installable with something like apt install qemu.

Problems with this approach:

• This is fragile, since there are other binaries in that package that this hack doesn't overwrite.
• This is impure, but I don't mind impurity in this case: after all, my fallback is to build an image and execute it with the host's QEMU anyway.

I'm happy to hear of simpler/more robust approaches to using the host's QEMU, while still using virtualisation.qemu.

[Pandapip1]

I'm in the process of packaging libcapsule, which might just be suitable.

[arximboldi]

Uhhhh, skimming through the README and some videos... this looks so interesting! Do you have some idea of how it could be used specifically to solve the libGL problem?

I am starting to dream of a plausible not so far future where we can use Nix to create portable .appimage for graphical applications... That would be really such a dream...

[vcunat]

Libcapsule indeed was a promising route attempted years ago. Hopefully it will go better now than back then. I see it got recent commits. 🤞🏽

[Pandapip1]

I am starting to dream of a plausible not so far future where we can use Nix to create portable .appimage for graphical applications... That would be really such a dream...

It's funny you should mention that use case. That's literally my use case as well!

---

Uhhhh, skimming through the README and some videos... this looks so interesting! Do you have some idea of how it could be used specifically to solve the libGL problem?

As I understand it, what libcapsule attempts to do is to create a library that at runtime, tries to use the system linker to find the actual libraries and expose them: https://people.collabora.com/~vivek/libcapsule.pdf

---

Thus far, I've managed to package libcapsule as well as make a simple builder to make libcapsule-wrapped libraries. The libcapsule derivation seems to be working fine, but something still needs to be fixed with the wrapper. I think I'm now at a point where it makes sense to draft a PR; I'll link that here: #337995

[Samueru-sama]

I am starting to dream of a plausible not so far future where we can use Nix to create portable .appimage for graphical applications... That would be really such a dream...

We were trying to do that here and just ran into this issue.