Support libGL in non-NixOS Nixpkgs

[matthewbauer]

Issue description

Trying to figure out some solutions for #9415, but wanted to see whether these ideas are workable. Feedback on this is appreciated!

The main goal here is to make libGL work out of the box on Nixpkgs without NixOS installed. This has been an issue for a long time, and could be fixed with adding LD_LIBRARY_PATH. I want this to work out of the box, though. I think there are two things we can do to improve this experience:

• Patch libglvnd to fallback to __GLX_VENDOR_LIBRARY_NAME=mesa when no system GL driver is found. This is software-based and slower, but at least it will work in a "pure" way. This could increase closure size, but I think those drivers might already be there.

• Add /usr/lib/libGL.so to the OpenGL RUNPATH. This is the location in almost every Linux distro, so it should be fairly predictable for us to use. This could be a libglvnd wrapped libGL though? This was already done for /run/opengl-driver in libglvnd, ocl-icd, vulkan-loader: Add /run/opengl-driver(-32) to RUNPATH. #60985. Ideally we could add /usr/lib/libGL.so without adding /usr/lib/ to avoid adding tons of impurities.

Discussed in https://discourse.nixos.org/t/software-rendering-using-mesa-not-possible/3023/8

/cc @ambrop72 @vcunat @edahlgren @abbradar

[ambrop72]

Isn't trying to load host libraries recipe for library version conflicts? I rather think we should have an easy way to install the correct driver libraries with nix.

[ambrop72]

I think a better solution is to make it so that if the user puts the correct driver in the user profile (e.g. nvidia-x11), the libraries should be found automatically. The implementation would involve some fiddling with dispatch libraries and driver config files (some config files for these things currently specify a library name only).

[matthewbauer]

Isn't trying to load host libraries recipe for library version conflicts? I rather think we should have an easy way to install the correct driver libraries with nix.

libGL should be backwards compatible, so binaries compiled with older version of libGL should still work with newer libGL versions. If the system libGL is older, this could be a problem, which is why we would also need a mesa fallback. This is how everyone outside of Nix world distributes binaries (with /usr/lib in the RUNPATH), so it's not exactly crazy. We'd definitely want it to be the lowest priority of all the RUNPATHs though.

I think a better solution is to make it so that if the user puts the correct driver in the user profile (e.g. nvidia-x11), the libraries should be found automatically. The implementation would involve some fiddling with dispatch libraries and driver config files (some config files for these things currently specify a library name only).

I don't think users should be messing with their libGL installation. It should be provided by the system in the same way the kernel or graphics server are. Also consider cases where we aren't installing anything. For instance, nix-bundle apps will not have a modifiable nix profile that a user can add to. We need a way for Nix closures to just work out of the box without installation.

[edahlgren]

@matthewbauer Thanks for opening the issue to discuss. Here's my understanding (echoing some points above):

1. Directly linking to host libraries is risky. libGL not working on non-NixOS (without setting up) #9415 mentions a libwayland conflict. And I think using system libs requires patchelf to fixup the rpaths. Do correct me if I'm wrong.

2. For direct rendering, there are several existing solutions for installing the correct drivers in the nix store. nvidia-x11 looks nice but seems like it's only for nvidia + linux? nixGL works for a wider range of configurations, but it requires the user to know their vendor and version number. To make this "just work" we'd probably need to write some scripts to detect the user's config and setup continuous integration for as many configurations as possible (os x graphics card), and add new test cases as people report that things don't work. For example, an edge case of nouveau mixed with nvidia was problematic for me when using nixGL for the first time.

3. Software rendering indeed is limited. But as a baseline / fallback option I think it's very important. In many cases it's actually fine to just use software rendering (e.g. code editors, simple qt apps, kcachegrind). __GLX_VENDOR_LIBRARY_NAME=mesa sounds like the best thing to use, but I need to thoroughly test it first.

4. Local indirect rendering is sort of a hack, but can be considered a last resort if the user can't get direct rendering to work. You'd symlink libGLX_indirect.so to a generic lib like libGLX_mesa.so (__GLX_VENDOR_LIBRARY_NAME=mesa might also work, need to test) and set LIBGL_ALWAYS_INDIRECT=1 in your environment. Requires an X server with the "+iglx" flag. It allows you to link your app to generic libs and send GLX requests over a local pipe or shared memory to your X server which is linked to your system libGL.so. It's one level of indirection for a clean nix / system boundary.

Am I missing a viable alternative?

[ambrop72]

The problem is not really libGL but all the other libraries that the libGL needs. It may expect a specific version of libstdc++, libdrm, etc.. For such libraries (assuming they are even found), either a library from the host system or from nix is going to be loaded first, and when some code from the other "world" needs it it will get the wrong library.

I think there is no solution to make this just magically work, without this risk with library versions. One thing that may look like it could work is dlmopen but is actually not useful for this use case with the current implementation (see https://sourceware.org/glibc/wiki/LinkerNamespaces).

[edahlgren]

@ambrop72 Beat me to the punch 😄

Yes, it's a frustrating situation. nix-bundle is in the same situation as AppImage IIUC. From the AppImage docs:

The ingredients used in your AppImage should not be built on a more recent base system than the oldest base system your AppImage is intended to run on. Some core libraries, such as glibc, tend to break compatibility with older base systems quite frequently, which means that binaries will run on newer, but not on older base systems than the one the binaries were compiled on.

But in nix, we want to give people flexibility to use whatever gcc / libc they want. That's a good thing. But the status quo isn't compatible with this.

In my fantasy world, I want libGL.so to be built into a separate process, that apps load and communicate with as a subprocess, so there's no version conflicts and you can use what's on your system. But that seems to be outside of the scope of this issue.

[Shados]

@ambrop72 I looked into dlmopen() too. I think it could form part of a solution, and further digging turned up libcapsule, which @dezgeg mentioned in #9415. It looks promising. There are still some limitations, mainly as a direct result of limitations in glibc's implementation of dlmopen(), but a solution built on top of libcapsule could likely work in nearly every case despite that.

[ambrop72]

I think that the only approach that would really solve the entire issue is to make the dynamic linker work in the following ways when dlopening a driver library from the host FHS world:

1. Loaded NixOS libraries do not satisfy dependencies of FHS libraries, except libc.
2. Loaded FHS libraries do not satisfy dependencies of NixOS libraries (only the library loaded from dlopen can be directly used).
3. FHS-based default library paths and ld.so.cache are used.

I think that getting #45105 to work could allow implementing this, possibly even without glibc code changes. The reason is that when libraries are linked by absolute path, they do not satisfy dependencies that are not absolute (not sure about the reverse, I hope that holds too). The libc exception could be implemented by not linking libc with an absolute path but still relying on RUNPATH.

[stale]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/help-understanding-the-libgl-abi-problem-and-possible-solutions/42022/9