Unwrapped GCC is propagated to user environments

[abbradar]

Issue description

Our CC wrapper doesn't wrap all the binaries that GCC provides -- namely prefixed with architecture x86_64-unknown-gnu-gcc and variations. Because unwrapped GCC is propagated to user environments from the wrapper it ends up in the path (for example in FHS environments). One example of problem that it causes is that OpenWrt picks it up and fails to compile.

wrapCC needs to detect and wrap those.

Steps to reproduce

1. buildEnv { paths = [ gcc ]; } (you can also add it to environment.systemPackages or buildFHSUserEnv);
2. Try to compile something with x86_64-unknown-gnu-gcc (prefix may vary).

Technical details

• Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion) 59dbcef

[collares]

Still relevant, I think. This might be interacting badly with newer versions of autoconf-archive macros AX_PROG_CC_FOR_BUILD and AX_PROG_CXX_FOR_BUILD, which are picking up unwrapped compilers and causing broken aarch64 builds for packages such as alsa-firmware (#126413, #126462, #143521) and singular. To see why this only affects aarch64, compare the following config.log excerpts for x86_64 (for which the unwrapped binary is x86_64-unknown-linux-gnu-gcc)...

configure:17568: checking for x86_64-pc-linux-gnu-gcc
configure:17603: result: no
configure:17613: checking for gcc
configure:17634: found /nix/store/xiq6j4jsyj351p8q3yw9cg1hdqp9m685-gcc-wrapper-10.3.0/bin/gcc
configure:17645: result: gcc
...
configure:18368: checking whether the C compiler works
configure:18390: gcc -g -O2   conftest.c  >&5
configure:18394: $? = 0
configure:18444: result: yes

...and aarch64...

configure:17568: checking for aarch64-unknown-linux-gnu-gcc
configure:17589: found /nix/store/919j2hfysp5iwvd2ssqkaj5n519pkaik-gcc-9.3.0/bin/aarch64-unknown-linux-gnu-gcc
configure:17600: result: aarch64-unknown-linux-gnu-gcc
...
configure:18368: checking whether the C compiler works
configure:18390: aarch64-unknown-linux-gnu-gcc -g -O2   conftest.c  >&5
/nix/store/5rs0jfwgffx4cizpsn7ffhhisl22bxh0-binutils-2.35.2/bin/ld: cannot find crt1.o: No such file or directory
/nix/store/5rs0jfwgffx4cizpsn7ffhhisl22bxh0-binutils-2.35.2/bin/ld: cannot find crti.o: No such file or directory
collect2: error: ld returned 1 exit status
configure:18394: $? = 1
configure:18434: result: no

which causes the following error in a project that uses AX_PROG_CC_FOR_BUILD:

checking whether the C compiler works... no
configure: error: in `/build/source/omalloc':
configure: error: C compiler cannot create executables
See `config.log' for more details

On aarch64, here are the directory contents:

$ ls /nix/store/919j2hfysp5iwvd2ssqkaj5n519pkaik-gcc-9.3.0/bin
aarch64-unknown-linux-gnu-c++  aarch64-unknown-linux-gnu-gcc        aarch64-unknown-linux-gnu-gcc-ar  aarch64-unknown-linux-gnu-gcc-ranlib  cpp  gcc     gcc-nm      gcov       gcov-tool
aarch64-unknown-linux-gnu-g++  aarch64-unknown-linux-gnu-gcc-9.3.0  aarch64-unknown-linux-gnu-gcc-nm  c++                                   g++  gcc-ar  gcc-ranlib  gcov-dump

$ ls /nix/store/lmjarab5jydvmfng8zcqp40bcq0141z3-gcc-wrapper-9.3.0/bin
as  c++  cc  cpp  g++  gcc  ld  ld.bfd  ld.gold  strip

This probably affects any package that uses the most recent versions of the AX_PROG_CC_FOR_BUILD and AX_PROG_CXX_FOR_BUILD autoconf macros. I don't understand enough about autoconf to know why the old versions didn't suffer from this, though, but here's the patch we apply to Singular to revert to older macro versions (maybe this particular bugfix is relevant?).

@Ericson2314, if you do have time could you give me some advice on how to make those new autoconf-archive macro versions not pick up unwrapped compilers? Thanks in advance, and sorry for cold-pinging!

EDIT: To be clear, although the problem involves AX_PROG_CC_FOR_BUILD, the failure above happens on a non-cross aarch64 build.

[L-as]

I think I've found a solution to this: 12cd91d#diff-1c3f080fce43dce74a94146b22cf2b6cb88f4a72c3491e82fef185393c152ff7R13

We just need CC_FOR_BUILD=$CC

[collares]

Thanks for taking a look at this!

Does this work for cross? If I understand correctly, CC_FOR_BUILD is supposed to be a compiler that produces binaries for the same architecture it runs on (that is, host == target for CC_FOR_BUILD), while $CC is not if you're cross-compiling. Setting CC_FOR_BUILD manually (perhaps to something else) looks like the way to go, though.

Edit: Indeed, adding "CC_FOR_BUILD=${buildPackages.stdenv.cc}/bin/cc" and "CXX_FOR_BUILD=${buildPackages.stdenv.cc}/bin/c++" to configureFlags (and ensuring buildPackages.stdenv.cc is in depsBuildBuild) seems to be an effective workaround for Singular.

[L-as]

That's a good question. It builds, but I actually haven't tested the cross-compiled binaries. I think it works though.

[Ericson2314]

CC_FOR_BUILD=$CC is a non-starter for cross, I am afraid.

[Ericson2314]

@collares I am a bit confused. IMO the unwrapped binaries being used is the problem. Those should be pretty "bannished" from showing up in Nix builds.

[Ericson2314]

All this makes me wish more I had time to fix #132343. We can then simplify cc-wrapper, which will generally help restore sanity. We might also be able to use the resulting freed up "complexity budget" to make a ccWithPacages like ghcWithPackages which would be good for various ad-hoc compilation outside nix derivations.

[collares]

Here's PATH for the build in question (this is for the singular derivation) as seen in config.log. Note that the wrapped gcc appears just before the unwrapped gcc. The original comment in this issue says "[b]ecause unwrapped GCC is propagated to user environments from the wrapper", and I took it at face value. I will check if this is singular-specific, though.

PATH: /nix/store/mk6jf2n2bysmgvyl3g3krx1ml8r2b74z-bison-3.8.2/bin/
PATH: /nix/store/32n29c24bipfj2igyk8nbkvl8fpkfnm3-gnum4-1.4.19/bin/
PATH: /nix/store/8sfnian5c4kxrn7375cpcd5vq42gzfhy-perl-5.34.0/bin/
PATH: /nix/store/wsh0gbw514d4d996ynfgnjm82ki0fq1m-pkg-config-wrapper-0.29.2/bin/
PATH: /nix/store/l6f1d3jwcpnd6y7arxipxlcn9w8kxaqz-autoconf-2.71/bin/
PATH: /nix/store/k3hsxr3j3k8zmb04mdmm3hjbflawb73a-automake-1.16.3/bin/
PATH: /nix/store/zvvajyh85j0rbwkzfq5xddap762r4qy6-gettext-0.21/bin/
PATH: /nix/store/fbkyvpjnxhh7kkgv3jy9r88lj3xhf7y2-libtool-2.4.6/bin/
PATH: /nix/store/rhcg1qsv8p05cqd7am2az69hmi2i79sh-sharutils-4.15.2/bin/
PATH: /nix/store/hg41y6yp6rzl5z0fikn11hp9xva6dgll-doxygen-1.8.20/bin/
PATH: /nix/store/5801xhsdi62mir8012ma8si0ms3f9bl5-graphviz-2.49.3/bin/
PATH: /nix/store/lgqzk1naj9gq72zb164ggm5m158bmywl-texinfo-4.13a/bin/
PATH: /nix/store/m2qvdhzhfj7wdglyicaq1s8zr3biq3z7-texlive-combined-small-2021.20210408/bin/
PATH: /nix/store/18biibvrl9dg65ixz322x1wlccmgvkrl-patchelf-0.13/bin/
PATH: /nix/store/lmjarab5jydvmfng8zcqp40bcq0141z3-gcc-wrapper-9.3.0/bin/
PATH: /nix/store/919j2hfysp5iwvd2ssqkaj5n519pkaik-gcc-9.3.0/bin/
PATH: /nix/store/1d3vmlyd1c26gbdfjf14ykmm9qdmm094-glibc-2.33-56-bin/bin/
PATH: /nix/store/00b76mg3gjj4k394psj85gxmb047w69h-coreutils-9.0/bin/
PATH: /nix/store/r7izxhclvhvsjagssq0mnwwfrymlgzmg-binutils-wrapper-2.35.2/bin/
PATH: /nix/store/5rs0jfwgffx4cizpsn7ffhhisl22bxh0-binutils-2.35.2/bin/
PATH: /nix/store/x4kj84z42zm9r55h37qrny7b561h7wd1-ncurses-6.2-dev/bin/
PATH: /nix/store/8dcsbcip7n6s0lka6qjab1r2xvzwvh82-ncurses-6.2/bin/
PATH: /nix/store/hk73bm8ghyd0a4dr19fqc49k2i0l14qn-readline-6.3p08/bin/
PATH: /nix/store/vfn4gl8ss50cxp1g48345v7sibx4pjms-lrcalc-1.2/bin/
PATH: /nix/store/a05p4npf1i14qvsg61w13qhy5iwz7xhx-gfan-0.6.2/bin/
PATH: /nix/store/m3j7jccj9jja8wm0b5wlipwk8si9dzh7-cddlib-0.94m/bin/
PATH: /nix/store/00b76mg3gjj4k394psj85gxmb047w69h-coreutils-9.0/bin/
PATH: /nix/store/7pwbf3i7sbr839sjrpp3slj2xs3cqzag-findutils-4.8.0/bin/
PATH: /nix/store/cpinvxbv2w0w8sl3xnfikkjnvr71s993-diffutils-3.8/bin/
PATH: /nix/store/mfkb1fqhr43zhn251ynf7zmx52h0mg6d-gnused-4.8/bin/
PATH: /nix/store/inysrik3db3a125vb1ivwg5f9lprb9pi-gnugrep-3.7/bin/
PATH: /nix/store/0ixdpwkdn2rsx7algazgkgvm5gkj7nd8-gawk-5.1.1/bin/
PATH: /nix/store/pz324z68c586mm9kp22pcdxxar9igxqx-gnutar-1.34/bin/
PATH: /nix/store/dp647s0x345iczbwrlcqjmyabcpxq7hv-gzip-1.11/bin/
PATH: /nix/store/y7nh0cjqdz7qydanzinpavqal4jb8in4-bzip2-1.0.6.0.2-bin/bin/
PATH: /nix/store/7a65i3akpsiwjq9xb8rr26m1rkx4g4wj-gnumake-4.3/bin/
PATH: /nix/store/9b7h9qbf4c7c236f53nmrz3x0g9cfj0p-bash-5.1-p12/bin/
PATH: /nix/store/9jnf0kjfgnyinagi44jh52vn7x2ipdhn-patch-2.7.6/bin/
PATH: /nix/store/z8hqh6bayrx3d6id0frrcwjva7261gpm-xz-5.2.5-bin/bin/
PATH: /home/collares/.nix-profile/bin/

[Ericson2314]

@collares is your config.log snippet from a nix build or trying to do thing with compilers from the user environment? I am confused.

[collares]

The error happens with a nix build, but to access log file I did nix develop nixpkgs#singular and ran genericBuild. If you want to reproduce the error, comment the AX_PROG_CC_FOR_BUILD revert patch in the Singular derivation (the config.log for the failing build is in the omalloc dir). I should have created a new issue to avoid confusion, sorry about that! I don't think the user environment matters in my case, but I could be wrong, just the part about the unwrapped compilers being exposed by the wrapper. Let me know if there's any info I can provide!

[collares]

I can see the unwrapped compiler on the path of any derivation. Let me know if I'm doing something wrong.

$ nix-shell --pure -E 'with (import ./. {}); pkgs.hello'
this path will be fetched (0.69 MiB download, 0.69 MiB unpacked):
  /nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz
copying path '/nix/store/3x7dwzq014bblazs7kq20p9hyzz0qh8g-hello-2.10.tar.gz' from 'https://cache.nixos.org'...

[nix-shell:~/nixpkgs]$ echo $PATH
/nix/store/yx24h6ywbkg7rm5x7lrm8mx8ypcf4ywi-bash-interactive-5.1-p8/bin:/nix/store/pjba17qx8vh7dln8m5vnkqdbbjahvg9b-patchelf-0.13/bin:/nix/store/xiq6j4jsyj351p8q3yw9cg1hdqp9m685-gcc-wrapper-10.3.0/bin:/nix/store/iyssx2arl8bc40pjimyfxyknj06swjdx-gcc-10.3.0/bin:/nix/store/q6prgn66s4achzrrnvcvyjgnm94c1bn3-glibc-2.33-56-bin/bin:/nix/store/l6f4z8mmcnnxba8w004xn28y0vr4gdkf-coreutils-9.0/bin:/nix/store/v7w0pspwq8r2b7k2sndxq3db843z7xm5-binutils-wrapper-2.35.2/bin:/nix/store/lbxfixyw1yk099pjyaiy3xj5dl7kxm1g-binutils-2.35.2/bin:/nix/store/l6f4z8mmcnnxba8w004xn28y0vr4gdkf-coreutils-9.0/bin:/nix/store/v3lvq9hqshyldc4i6f5jy0zs0k5psbws-findutils-4.8.0/bin:/nix/store/gxj3wkwc5m5vb8b3rv6h029ky9nan4bf-diffutils-3.8/bin:/nix/store/vklvyr82ajbz7jm7g8dbkh62k20b0dpr-gnused-4.8/bin:/nix/store/nkwls56wcfwi1r0jnkqkvwx2zk7w3qrz-gnugrep-3.7/bin:/nix/store/21nzalbq7ay1vzanri1dl4845s3cl72i-gawk-5.1.1/bin:/nix/store/wny483nz9q52xq8azwpqna3pq98zcsgy-gnutar-1.34/bin:/nix/store/mp7lwrwr0nl69rznln13j3k4zf69s80i-gzip-1.11/bin:/nix/store/m9jxbbbxfzi6052yw3kgz95wvz76mbb1-bzip2-1.0.6.0.2-bin/bin:/nix/store/315glp1si526x60nhhkbqmip4m7b25a7-gnumake-4.3/bin:/nix/store/a54wrar1jym1d8yvlijq0l2gghmy8szz-bash-5.1-p12/bin:/nix/store/h2xzp4a764hn5n9br63ilh9qglz3baxn-patch-2.7.6/bin:/nix/store/3q4xj02ijqgfxffndvg28nlv741sx6qx-xz-5.2.5-bin/bin

[nix-shell:~/nixpkgs]$ NIX_DEBUG=1 x86_64-unknown-linux-gnu-gcc
x86_64-unknown-linux-gnu-gcc: fatal error: no input files
compilation terminated.

[Ericson2314]

@collares so the weird thing is how does the autoconf macro somehow get the prefixed gcc which should be "shadowed" by the wrapper coming earlier in the PATH.

[collares]

There's no wrapper for the prefixed gcc here, just for the unprefixed one:

$ ls /nix/store/xiq6j4jsyj351p8q3yw9cg1hdqp9m685-gcc-wrapper-10.3.0/bin
as  c++  cc  cpp  g++  gcc  ld  ld.bfd  ld.gold  strip

[nix-shell:~/nixpkgs]$ ls /nix/store/iyssx2arl8bc40pjimyfxyknj06swjdx-gcc-10.3.0/bin
c++  cpp  g++  gcc  gcc-ar  gcc-nm  gcc-ranlib  gcov  gcov-dump  gcov-tool  lto-dump  x86_64-unknown-linux-gnu-c++  x86_64-unknown-linux-gnu-g++  x86_64-unknown-linux-gnu-gcc  x86_64-unknown-linux-gnu-gcc-10.3.0  x86_64-unknown-linux-gnu-gcc-ar  x86_64-unknown-linux-gnu-gcc-nm  x86_64-unknown-linux-gnu-gcc-ranlib

On x86_64 this doesn't matter because the macro searches for the wrong prefix (x86_64-pc-linux-gnu instead of x86_64-unknown-linux-gnu) and then tries the unprefixed version, but on aarch64 the macro finds the unwrapped prefixed compiler on the first try.

[L-as]

I am not familiar with how autoconf works, but to me it looks like it tries looking for the compilers in $PATH in alphabetic order, meaning aarch64... comes before cc or gcc. The same does not hold for x86....
Of course this may be very wrong.

[L-as]

Ah, it's probably what collares says.

[Ericson2314]

There's no wrapper for the prefixed gcc here, just for the unprefixed one

Why is this? We should always be using wrapped compilers in nix builds.

[collares]

I don't think I have any good explanation here, but I would be happy to run any tests you think would be useful. Three things seem to be happening:

• My stdenv does not contain wrappers whose names are prefixed. This is also the case on the aarch64 build that failed.
• The directory containing the unwrapped binaries ends up in PATH in the nix build (after the wrapper directory, but still).
• The newer version of the AX_PROG_CC_FOR_BUILD macro guesses a prefixed binary name (aarch64-unknown-linux-gnu-gcc) and the only thing on the PATH matching that name is the unwrapped version.

The end result is that the macro ends up finding an unwrapped compiler. The macro was only updated last year (in autoconf-archive/autoconf-archive#207), and the first version of autoconf-archive that contains the updated version was released this year, so not a lot of packages have updated yet, and the number of packages that use CC_FOR_BUILD is probably not too large.

[collares]

After reading https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/cc-wrapper/default.nix#L36, I think I see a possible source of confusion. To clarify, my comment above (#21751 (comment)) does not refer to a cross build: The build that failed on aarch64 was supposed to generate aarch64 binaries. The x86_64 examples are on my own laptop, and the aarch64 examples are being run on Oracle Cloud.

[Ericson2314]

Hmm so the issue is that the native unwrapped binaries are both prefixed and unprefixed, but the wrapped ones are only unprefixed? CC @sternenseemann who was looking down exactly when GCC is prefixed to help dodge this sort of thing.

[collares]

Yep, that's the issue! This wasn't a problem before, but with the new autoconf macros it is.

[sternenseemann]

We force the prefix to be the same as what nix thinks targetPrefix should be in general. So I guess we need to figure out why CC_FOR_BUILD would get a prefix for a native build?

[Ericson2314]

@sternenseemann I am not sure CC_FOR_BUILD is set at all in this case, I think autoconf is just guessing names, tries with the prefix first, and then slips passed the wrapped one to find the unwrapped one. But I am not sure.

[collares]

Yes, exactly, CC_FOR_BUILD is not set on the corresponding nix develop shell, since the derivation didn't have the native compiler in depsBuildBuild. Arguably this is a problem with the derivation, since depsBuildBuild should be set if the build will require something that is definitely a native compiler. But the fact that post-autoconf-archive/autoconf-archive#207 autoconf finds the prefixed (and unwrapped) version automatically on a native build and then fails to generate executables with it leads to very confusing error messages, especially since things work fine on x86_64 (autoconf guesses the wrong prefix there) but fail on aarch64.

[collares]

If it helps, I have confirmed that autoconf-archive/autoconf-archive@9c26407 is the particular commit that causes the AX_PROG_CC_FOR_BUILD macro to guess the prefixed name for this configure.ac. ./configure is called with the following flags: --disable-option-checking --prefix=/home/collares/nixpkgs/outputs/out --disable-static --with-ntl=/nix/store/hrwzsb11xf9w7saygfjdczf2mqff62il-ntl-11.5.1 --disable-pyobject-module --enable-doc-build --enable-gfanlib CC=gcc CXX=g++ --enable-omalloc OMALLOC_LIBS=/home/collares/source/omalloc/libomalloc.la OMALLOC_INCLUDES=-I/home/collares/source --with-Singular RESOURCES_LIBS=/home/collares/source/resources/libsingular_resources.la 'RESOURCES_INCLUDES=-I/home/collares/source ' FACTORY_LIBS=/home/collares/source/factory/libfactory.la 'FACTORY_INCLUDES=-I/home/collares/source -I/home/collares/source/factory -I/home/collares/source/factory/include' --cache-file=/dev/null --srcdir=.

By the way, is it possible for $build to be empty? I thought it always referred to the build arch (unless the arch is super bizarre and can't be detected), even if --build wasn't passed explicitly as a command line argument. If not, then it seems clear that this commit really tries the prefixed compiler, because it does pushdef([CC], CC_FOR_BUILD) (when CC_FOR_BUILD is empty), pushdef([ac_tool_prefix], ac_build_tool_prefix) and ac_build_tool_prefix="$build-", so this is going to try the prefixed compiler.

[Ericson2314]

Yeah I put no blame on autoconf. We simple should make sure that no compiler, unprefixed otherwise, can
"leak" past the wrapper earlier the PATH.

Put simply, if the unwrapped provides prefixed, the wrapped should provide prefixed, and if the unwrapped provides unprefixed, the wrapped should also provide unprefixed.

[sternenseemann]

I think we just need to prevent gcc-unwrapped from creating prefixed compilers if --program-prefix="". targetPrefix should be accurate and the only way to find the compiler. It seems like as a quick fix we could delete all symlinks in gcc's $out/bin, although I'm a bit confused by ${stdenv.targetPlatform.config}-gcc-{ar, nm, ...} which seem to be proper binaries?

[Ericson2314]

That makes sense, thanks for taking a look. I guess we can rename those but also see why they were make like that in the first place.

[sternenseemann]

It creates both gcc-ar and x86_64-unknown-linux-gcc-ar for some reason, whereas x86_64-unknown-linux-gcc is just a symlink to gcc.

[collares]

https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/gcc/builder.sh#L279