Any way for LD_LIBRARY_PATH to *not* break nix-env's absolute paths for dynamic libs

[rrnewton]

Maybe this is naive of me, but I thought there was some way on linux to link a dynamic library with an absolute path which would not be affected by LD_LIBRARY_PATH.

I'm on an ubuntu 14.04 system on a shared account where, for whatever reason, LD_LIBRARY was set to include some extra directories. That causes nix-env to segfault:

$ nix-env -q
Segmentation fault (core dumped)
$ ldd `which  nix-env`
        linux-vdso.so.1 =>  (0x00007ffde2da7000)
        libnixexpr.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixexpr.so (0x00007f72a54ac000)
        libgc.so.1 => /nix/store/f24zx1r39kalz01q9kw7zcg1ngj7w2db-boehm-gc-7.2f/lib/libgc.so.1 (0x00007f72a5243000)
        libnixmain.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixmain.so (0x00007f72a5031000)
        libnixstore.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixstore.so (0x00007f72a4d6f000)
        libnixutil.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixutil.so (0x00007f72a4b3d000)
        libnixformat.so => /nix/store/9n8c3g541qn43yjjs94f1a0m69wp8scg-nix-1.11.2/lib/libnixformat.so (0x00007f72a4933000)
        libstdc++.so.6 => /nix/store/kxac9yv62zff2pj6k1j4d6hiz8jdi2f3-gcc-5.3.0-lib/lib/libstdc++.so.6 (0x00007f72a45b9000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f72a42b3000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f72a409c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f72a3cd7000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f72a3ad3000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f72a38b5000)
        /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x0000$f72a573a000)
        libsqlite3.so.0 => /nix/store/1w64w28j9vz57g2k5bivh2wpk8q9j6vj-sqlite-3.12.2/lib/libsqlite3.so.0 (0x00007f72a35e0000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f72a33d0000)
        libcurl.so.4 => /nix/store/9whcgafma7473b4xpyb88hbhi8w5414z-curl-7.47.1/lib/libcurl.so.4 (0x00007f72a3160000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f72a2f3e000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f72a2b62000)
        libnghttp2.so.14 => /nix/store/vv2n25vgck007mjph2ay8g4ab58igzp2-nghttp2-1.9.2-lib/lib/libnghttp2.so.14 (0x00007f72a293$000)
        libssh2.so.1 => /nix/store/y8jcz5g52265i18kr63d6lim4k1d7jgn-libssh2-1.7.0/lib/libssh2.so.1 (0x00007f72a2711000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f72a24b2000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f72a2299000)

Whereas simply unsetting LD_LIBRARY_PATH makes nix-env run fine:

$ LD_LIBRARY_PATH= nix-env -q
bash-4.3-p42

That's an easy fix, sure. But shouldn't nix-env be robust against LD_LIBRARY_PATH if possible?

[rrnewton]

I'm not even sure what it means when ldd reports <absolute path> => <absolute path> in its output. That's what it does for ld-linux-x86-64.so.

Likewise, here's the ldd printout for bash, this time without any LD_LIBRARY_PATH shenanigans:

   $ ldd /nix/store/d20f169ryps7ds2qak0r5n1f4hhxr80h-bash-4.3-p42/bin/bash
    linux-vdso.so.1 =>  (0x00007fffa6a72000)
    libdl.so.2 => /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/libdl.so.2 (0x00007f0bb1705000)
    libc.so.6 => /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/libc.so.6 (0x00007f0bb1363000)
    /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f0bb1909000)

readelf -d makes it look like libc.so is in fact a bare name, not an absolute path:

Dynamic section at offset 0xb0020 contains 26 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

What I wanted to see was the nix binary using only its own ld-linux.so and pointing to its own libs exclusively, with absolute paths. Since bash is a core package built from source, it should specify these linker options in the derivation (and not need PatchElf), right?

[vcunat]

This belongs to nixpkgs, really, but let's leave it here.

The way we've always done this:

• we set the loader to an absolute path (that's the ld-linux*.so*),
• the usual short so-names are used, e.g. libdl.so.2, and they're searched in a list of absolute RPATHs coded into the ELFs;
• $LD_LIBRARY_PATH can be used to add directories where to search before RPATH (IIRC we use a nixpkgs-specific patch for the loader to change the priority here).

Recently there was a suggestion on ML to use absolute-path so-names instead of RPATH, but I've seen no code yet and I suspect that way might be (significantly) more complicated to implement than what we're doing nowadays.

[rrnewton]

Ok, so it sounds like it's a known problem that LD_LIBRARY_PATH can break executables in /nix/store.

I spent some time reading about rpath, but I wasn't having luck controlling rpath or the loader directly from my own nix build (where I'm going through gcc-wrapper).

If we're always running a custom load, what the heck does it mean that ld-linux-x86-64.so claims to point to an Ubuntu system version under /lib64? E.g.:

    /nix/store/phffgv3pwihmpdyk8xsz3wv8ydysch8w-glibc-2.23/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f0bb1909000)

[vcunat]

If we're always running a custom load, what the heck does it mean that ld-linux-x86-64.so claims to point to an Ubuntu system version under /lib64?

I'd think that ldd shows this one wrong. IIRC running ldd from a different Linux can show information different from what really happens during running. Best use ldd from nixpkgs.

[copumpkin]

@rrnewton just because it's known doesn't mean it's good or that we want it to persist. Is there another ticket tracking this issue? If not, I'd just leave it open.

[vcunat]

There are some places where we rely on $LD_LIBRARY_PATH. We would have to adapt those.

$ git grep LD_LIBRARY_PATH | wc -l
271

[rrnewton]

@copumpkin, yes indeed it would be great to fix it. I really want to be able to tell people "run this command; it's nix so I can guarantee it will do the same thing for you as for me". But in a multi-user environment sitting on top of another distro... we just haven't fully been able to realize that dream. This is one the things we trip over.

[vcunat]

I don't think distros normally define $LD_LIBRARY_PATH by default. (Well, NixOS does.) On non-NixOS there's also often problem with libGL, which is related.

Really, it's difficult to balance what the user wants there, e.g. we've been arguing whether to use /etc/foo from the host OS – IMHO basically noone wants nix stuff completely disregard config from the host machine (includes also ~/.* and env vars).

[rrnewton]

Yeah, but on shared machines where people can't install in /usr then users end up with libraries installed in their home directories and then they set their LD_LIBRARY_PATH to make it work. This is probably a terrible thing. But there have to be some non-zero number of people in the world who set an LD_LIBRARY_PATH in their bashrc or profile.

I'm not sure what other people want regarding the isolation of nix on other distros, but whenever we've "mixed" the two worlds, things go south really fast. (E.g. install a compiler via nix and build something outside of nix with it.)

[rrnewton]

I'm perfectly happy if the manual is just updated to say loudly and clearly "unset LD_LIBRARY_PATH before calling any Nix-built binary".

[vcunat]

Yeah, we could document that.

[amosbird]

Can I add a post install hook that change all RUNPATH to RPATH? I have customized toolchain that relies on LD_LIBRARY_PATH while I still want to use binaries from nix-env like cmake and ninja.

[vcunat]

@amosbird: I think the easiest way for you is to wrap the cmake/ninja executables to unsed that variable for them.

[amosbird]

@vcunat Isn't that variable masking going to be inherited by subprocesses like toolchain binaries using LD_LIBRARY_PATH?

[vcunat]

I firmly believe that (exported) variables are inherited in the state you have them, so if you unset/change some, they'll be inherited that way.

[amosbird]

@vcunat yes, and that will break the toolchain's binaries.

[vcunat]

"that way" means unset, i.e. the way almost everyone uses the toolchain.

[vcunat]

Ah, and you want it the other way, probably. Linkify: #16767

[stale]

I marked this as stale due to inactivity. → More info

[stale]

I closed this issue due to inactivity. → More info

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/glibc-2-33-not-found-although-it-is-installed/14516/3

[Atry]

I think it's harmless to patch glibc to introduce an environment variable to specify the fallback library path other than LD_LIBRARY_PATH. The new environment variable should be used to search after runpath.

[jeff-hykin]

I don't think anything that extreme is necessary. The rpath already takes priority over the LD_LIBRARY_PATH and the runpath. If nix binaries patch their rpath with the correct paths using patchelf, then AFAIK they wouldn't be effected by LD_LIBRARY_PATH changes.

Alternatively, nix binaries could easily be wrapped in a shell script that clears, or allows setting a binary-specific LD_LIBRARY_PATH.

For reproductibility I think it's really important that nix binaries find the shared objects they were built with FIRST and then let something like the LD_LIBRARY_PATH be a fallback option.

I'm not sure why it seems neither of these options are used.

[Atry]

Setting LD_LIBRARY_PATH would break Ubuntu executables even if nixpkgs switches to rpath. See cachix/devenv#773

[Atry]

Wrapper script is not an ideal solution.
If LD_LIBRARY_PATH is set in a wrapper of python to locate libstdc++, python will be able to load a package in manylinux ABI. However, any Python code to start a new subprocess will fail if the subprocess was compiled with a different version of glibc or libstdc++, no matter if the executable of the subprocess is from Ubuntu or nixpkgs.

[jeff-hykin]

Wrapper script is not an ideal solution.

Agreed, for exactly the reason you mentioned (subprocess) I just meant it's still better than the suggested "solution" above to manually unset LD_LIBRARY_PATH before calling a nix binary. That^ solution could, at minimum, be automated with wrappers.

Setting LD_LIBRARY_PATH would break Ubuntu executables

I mean I don't disagree but setting the rpath means nix won't need/care about the LD_LIBRARY_PATH. We could have:

• nix binarys work (always, using rpath), Ubuntu binaries don't (when we set LD_LIBRARY_PATH for some non-nix reason)
• nix binarys don't work when we set LD_LIBRARY_PATH, and Ubuntu binaries don't work when we set LD_LIBRARY_PATH (current system)

Nix can't really control the Ubuntu binaries, so it seems like the rpath solution is still the way to go.

[Atry]

#248547 would fix this issue

[jeff-hykin]

NixOS/nixpkgs#248547 would fix this issue

I don't think so, if we want reproducable behavior then the linked-libraries can't be part of my-machines coincidentally what-env-vars-I-have-at-the-moment.

Adding another ENV var just makes runtime behavior even more complex and difficult reproduce.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/envfs-fix-shebangs-on-nixos/11234/2

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/envfs-fix-shebangs-on-nixos/11234/4

[SomeoneSerge]

#248547 would fix this issue

Only partially, I believe. There's something fundamentally problematic with DT_RUNPATH's priority, and we'd still suffer e.g. from #287764, because apparently some distributions do set LD_LIBRARY_PATH

DT_RPATH

Is deprecated upstream. We should collect and summarize all of the issues we've been having with DT_RUNPATH and LD_LIBRARY_PATH, and open a ticket in glibc support.

I don't think distros normally define $LD_LIBRARY_PATH by default. (Well, NixOS does.)

Just for future readers: at least in year 2023 (the quote is from 2016 and idk how it was back then) NixOS does not set LD_LIBRARY_PATH, not out of the box. Some of the wrappers in Nixpkgs do extend LD_LIBRARY_PATH, which is a different story.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/on-nixpkgs-and-the-ai-follow-up-to-2023-nix-developer-dialogues/37087/2

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/library-not-found-with-libc-so-6-version-glibc-2-34-not-found/45339/1

[DDoSolitary]

FWIW, I took the idea of using LD_AUDIT modules mentioned in #248547 (comment) and wrote an audit module to force ld.so to search DT_RUNPATH before LD_LIBRARY_PATH

https://github.com/DDoSolitary/ld-audit-prefer-runpath