lld_14 with clang_14 cannot use libstdc++.so.6: cannot open shared object file: No such file or directory

[inclyc]

Describe the bug

llvm-tblgen can not find libstdc++.so.6.

FAILED: include/llvm/IR/Attributes.inc <llvm-project>/llvm-project/build/include/llvm/IR/Attributes.inc 
cd <llvm-project>/llvm-project/build && <llvm-project>/llvm-project/build/bin/llvm-tblgen -gen-attrs -I <llvm-project>/llvm-project/llvm/include/llvm/IR -I<llvm-project>/llvm-project/build/include -I<llvm-project>/llvm-project/llvm/include <llvm-project>/llvm-project/llvm/include/llvm/IR/Attributes.td --write-if-changed -o include/llvm/IR/Attributes.inc -d include/llvm/IR/Attributes.inc.d
<llvm-project>/llvm-project/build/bin/llvm-tblgen: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

Confirmed that g++/clang++ can generate correct minimal C++ binary (with libstdc++), but recently llvm-tblgen breaks.

Steps To Reproduce

Steps to reproduce the behavior:

1. Use recent nixpkgs/nixos-unstable
2. build the llvm-project

LLVM Revision: d768b97424f9e1a0aae45440a18b99f21c4027ce

Flags:

LLVM=~llvm-project

llvm_base="$LLVM"
build_mode="Debug"
build_dir="build"
build_base="${llvm_base}/${build_dir}"
llvm_enable_projects="clang;mlir"


cmake -DBUILD_SHARED_LIBS:STRING=On \
-DCMAKE_BUILD_TYPE:STRING=${build_mode} \
-DLLVM_APPEND_VC_REV:STRING=Off \
-DLLVM_ENABLE_LLD:STRING=On \
-DLLVM_ENABLE_PROJECTS:STRING="${llvm_enable_projects}" \
-DLLVM_LINK_LLVM_DYLIB:STRING=Off \
-DLLVM_OPTIMIZED_TABLEGEN=Off \
-DLLVM_CCACHE_BUILD:BOOL=FALSE \
-DLLVM_USE_SPLIT_DWARF=On \
-DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE \
-DCMAKE_C_COMPILER:FILEPATH=${CC:-"gcc"} \
-DCMAKE_CXX_COMPILER:FILEPATH=${CXX:-"g++"} \
-S${llvm_base}/llvm \
-B${build_base} -G Ninja \
-DLLVM_TARGETS_TO_BUILD="host;RISCV;"

Expected behavior

Compile LLVM successfully as before

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.9, NixOS, 23.05 (Stoat), 23.05.20230206.5a350a8`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.2`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[inclyc]

According to some of my local tests, -DLLVM_ENABLE_LLD is important to reproduce. BFD linker gives me other warnings:

[user@system:~]$ cmake --build build
[313/5616] Linking CXX executable bin/not
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: librt.so.1, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: libdl.so.2, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
[328/5616] Linking CXX executable bin/UnicodeNameMappingGenerator
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: librt.so.1, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: libdl.so.2, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
[331/5616] Linking CXX executable bin/split-file
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: librt.so.1, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: libdl.so.2, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
[333/5616] Linking CXX executable bin/yaml-bench
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: librt.so.1, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: libdl.so.2, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
[529/5616] Linking CXX executable bin/FileCheck
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: librt.so.1, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: libdl.so.2, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
[553/5616] Linking CXX executable bin/llvm-tblgen
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: librt.so.1, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: libdl.so.2, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
[594/5616] Linking CXX executable bin/clang-tblgen
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: librt.so.1, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)
/nix/store/r2b9k28c6aghczpqfvh71y9xavm7rr68-binutils-2.39/bin/ld: warning: libdl.so.2, needed by lib/libLLVMSupport.so.17git, not found (try using -rpath or -rpath-link)

[inclyc]

CC @Ericson2314

[inclyc]

Reduced repro:

#include <iostream>

int main() {
    std::cout << "Test" << std::endl;
}

[user@system:~]$ clang++ -fuse-ld=lld test.cpp -o test
[user@system:~]$ ./test
./test: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
[user@system:~]$ which lld
/run/current-system/sw/bin/lld
[user@system:~]$ ll `which lld`
lrwxrwxrwx 3 root root 62 Jan  1  1970 /run/current-system/sw/bin/lld -> /nix/store/k1kpcazajfi8k2lkpnf98idxapxdbhf0-lld-14.0.6/bin/lld*
[user@system:~]$ ll `which clang`
lrwxrwxrwx 3 root root 74 Jan  1  1970 /run/current-system/sw/bin/clang -> /nix/store/1bmrs2b76lzzn90zj2zg6049fxii7g3z-clang-wrapper-14.0.6/bin/clang*

[ocfox]

Working fine with following shell.

with import <nixpkgs> { };
pkgs.clangStdenv.mkDerivation {
  name = "test";
  nativeBuildInputs = with pkgs; [
    lld
  ];
}

[inclyc]

Confirmed that this regression was introduced between 2caf4ef and 0699920

[inclyc]

Working fine with following shell.

with import <nixpkgs> { };
pkgs.clangStdenv.mkDerivation {
  name = "test";
  nativeBuildInputs = with pkgs; [
    lld
  ];
}

I made a nix flake that reproduces this problem:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils, ... }@inputs:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
      in
      {
        devShells.default = pkgs.mkShell
          {
            nativeBuildInputs = with pkgs; [
              clang_14
              lld_14
            ];
          };
      });
}

https://gist.github.com/inclyc/8df38510fd141965da238c6d4d6edeca

[inclyc]

CC maintainers: @Ericson2314 @sternenseemann @lovek323 @dtzWill @primeos

[trofi]

Does staging-next branch work for you? Chances are it is fixed by #213185

[inclyc]

Does staging-next branch work for you? Chances are it is fixed by #213185

Happy to test whether the current staging-next branch nixpkgs would solve the issue. However, after I switch to staging-next, lld_14 and clang_14 need to compile a lot of packages when I tried to enter nix develop shell. Is there any way to speed up the compilation? The solutions I think of are: increase the number of parallel compilations? , looking for binaryCache for staging-next?

increase the number of parallel compilations

nix develop compiles the required dependencies almost sequentially, and a lot of time is spent on "configuring xx, checking for xxx" on a single core. Is there any way to increase the scale of parallelism? (I've set big-parallel option).

[trofi]

nix develop -j <N> should increase job parallelism. nix develop --cores <N> should increase build parallelism within a job. If you want to build less you can sync to latest commit hydra already built for staging-next. For https://hydra.nixos.org/jobset/nixpkgs/staging-next looks like 123c88b is reasonably complete.

[inclyc]

Thanks! @trofi

Confirmed that the problem has been solved in 123c88b.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/123c88bcd0acd32c1d224187fd27cc9001b0e1a8";
  };

  outputs = { self, nixpkgs, ... }@inputs:
    {
      nixosConfigurations.container = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules =
          [
            ({ pkgs, ... }: {
              boot.isContainer = true;
              environment.systemPackages = with pkgs; [
                clang_14
                lld_14
              ];
            })
          ];
      };
    };
}