GCC stdenv broken when cross-compiling to same system and depsBuildBuild contains buildPackages.stdenv.cc

[lilyball]

Describe the bug

If I have a package that is configured with depsBuildBuild = [ buildPackages.stdenv.cc ], this package will fail to compile when building with the GCC stdenv and cross-compiling back to the same system. The simplest way to see this is to build something that uses gccStdenv in the pkgsLLVM set, such as pkgsLLVM.kexec-tools, but this will affect other packages too (such as ncurses) if I do a custom cross-compile setup like

crossSystem = {
  system = builtins.currentSystem;
  dummyValueForCrossCompiling = true;
};

The rationale for such a setup is I'm using crossOverlays to swap out glibc for the host system, and I need the crossSystem definition or stdenv just uses the stage 2 glibc instead.

The root cause seems to be that unwrapped GCC always includes target-prefixed binaries even when not cross-compiling.

A quick grep through nixpkgs suggests this issue will affect close to 100 derivations.

Steps To Reproduce

nix build nixpkgs/5ba549eafcf3e33405e5f66decd1a72356632b96#pkgsLLVM.kexec-tools

Build log

last 10 log lines:
> updateAutotoolsGnuConfigScriptsPhase
> Updating Autotools / GNU config script to a newer upstream version: ./config/config.sub
> Updating Autotools / GNU config script to a newer upstream version: ./config/config.guess
> configuring
> configure flags: --prefix=/nix/store/91xaishiyx0q54ljmlppxay2sv4kgn9v-kexec-tools-x86_64-unknown-linux-gnu-2.0.26 BUILD_CC=cc --build=x86_64-unknown-linux-
gnu --host=x86_64-unknown-linux-gnu
> checking for x86_64-unknown-linux-gnu-gcc... x86_64-unknown-linux-gnu-gcc
> checking whether the C compiler works... no
> configure: error: in `/build/kexec-tools-2.0.26':
> configure: error: C compiler cannot create executables
> See `config.log' for more details

If I keep the failed build and inspect the resulting config.log file it fails at

configure:2467: checking whether the C compiler works
configure:2489: x86_64-unknown-linux-gnu-gcc    conftest.c  >&5
/nix/store/rhhll3vwpj38ri72ahrrrvcbkhz4fhh6-binutils-2.40/bin/ld: cannot find crt1.o: No such file or directory
/nix/store/rhhll3vwpj38ri72ahrrrvcbkhz4fhh6-binutils-2.40/bin/ld: cannot find crti.o: No such file or directory
/nix/store/rhhll3vwpj38ri72ahrrrvcbkhz4fhh6-binutils-2.40/bin/ld: cannot find -lgcc_s: No such file or directory
collect2: error: ld returned 1 exit status
configure:2493: $? = 1
configure:2531: result: no

I dug into what's going on here and the problem is that the $PATH for such a derivation ends up looking like

"${buildPackages.stdenv.cc}/bin:${buildPackages.stdenv.cc.cc}/bin:[…]:${buildPackages.stdenv.cc.bintools}/bin:${buildPackages.stdenv.cc.bintools.bintools}/bin:[…]:${stdenv.cc}/bin:${stdenv.cc.cc}/bin:[…]:${stdenv.cc.bintools}/bin:${stdenv.cc.bintools.bintools}/bin:[…]"

Notice here how the buildPackages.stdenv wrapped compiler and unwrapped compiler show up before the stdenv wrapped compiler.

buildPackages.stdenv.cc contains only unprefixed binaries (e.g. gcc), because it only includes the prefix when hostPlatform != targetPlatform and buildPackages.stdenv uses buildPackages.buildPackages.gcc. However the unwrapped compiler does include the target-prefixed binaries. This means that when the package tries to look for $CC it finds the prefixed binary in the unwrapped build compiler instead of finding the expected cross-compiler.

It looks to me like this setup was designed with the expectation that cross-compiling will always go to another system and therefore the target-prefixed binary will always uniquely refer to the host compiler (is that the right name for it?), but when cross-compiling back to the same system it ends up being the unwrapped build compiler instead.

This isn't noticed for most of the packages in pkgsLLVM because the host compiler is clang instead of gcc, it's only packages that use gccStdenv that have the problem, or it's when cross-compiling without setting useLLVM.

Based on this, it seems the root cause is that the unwrapped GCC always includes target-prefixed binaries even when not cross-compiling. Clang doesn't do this (nor does bintools), so I don't understand why GCC does. Because it does that, it means that any derivation that looks for {target}-gcc is going to get a broken compiler.

I noticed in cc-wrapper/default.nix there's a comment suggesting that cc-wrapper should always include the target-prefixed binaries:

nixpkgs/pkgs/build-support/cc-wrapper/default.nix

Lines 74 to 79 in fdc5d61

	# Prefix for binaries. Customarily ends with a dash separator.
	#
	# TODO(@Ericson2314) Make unconditional, or optional but always true by
	# default.
	targetPrefix = lib.optionalString (targetPlatform != hostPlatform)
	(targetPlatform.config + "-");

This would mask the issue but cause another one. It would mean that {target}-gcc resolves to the build compiler instead of the host compiler. It would be a wrapped compiler, so compilation would work, but it will use the build linker instead of the host linker (which matters for e.g. pkgsLLVM) and it will use the build libc instead of the host libc (which matters for me because I'm swapping out glibc with crossOverlays).

Notify maintainers

@Synthetica9 @vcunat @Ericson2314 @amjoseph-nixpkgs

Metadata

❯ nix run nixpkgs#nix-info -- -m
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.51, NixOS, 23.11 (Tapir), 23.11.20230919.5ba549e`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - nixpkgs: `/etc/nix/inputs/nixpkgs`

[ghost]

I think this is a duplicate of

• pkgsCross.hostPlatform.bash fails to build if buildPlatform.config==hostPlatform.config but buildPlatform!=hostPlatform #264989

There is a test case for that bug in

• lib.systems, test.cross.sanity: add test case for #264989 #264992

and a fix in

• gcc.libgcc: compare host and target platforms, rather than their triples #264976

[ghost]

See

1c85a4c

[ghost]

buildPackages.stdenv.cc is unspliced. That's the real problem.

BTW it is not your fault for not noticing this.

The splicing mechanism in nixpkgs is simply batshit-insane. I've given up on trying to rally support for replacing/removing it, and am settling for adding warnings (e.g. #265121) to all the situations where it does things that are totally unexpected.

[lilyball]

Don't do that. buildPackages.stdenv.cc is unspliced. That's the real problem.

I'm not doing it, packages I'm trying to use are doing it (such as ncurses). Also this approach is documented in the manual.

Regarding 1c85a4c0272e2c4a63f82813953642abc5cdc37c, won't setting depsBuildBuild = [ stdenv ]; be equivalent to depsBuildBuild = [ pkgsBuildBuild.stdenv ]? The splicing stuff doesn't have any special-case for stdenv so it's going to get the offset wrong. As near as I can tell, depsBuildBuild = [ buildPackages.stdenv.cc ] is perfectly correct, because buildPackages.stdenv.cc is the compiler used to build things in buildPackages, and derivations are doing this when they need to build something that they then invoke as part of the build process. If what they're building actually cares about the target config then invoking pkgsBuildBuild.stdenv.cc will be wrong as it will use the build platform as its target.

Incidentally, you kept linking this issue instead of the PR you're talking about that adds warnings.

[lilyball]

Actually I think I overcomplicated it. Using depsBuildBuild = [ stdenv ] will get pkgsBuildBuild.stdenv, which is indeed incorrect as it has the wrong target platform, but these derivations just want the compiler so the platforms attached to the stdenv don't really matter as long as the compiler is correct. buildPackages.stdenv.cc == pkgsBuildBuild.gcc (absent overrides), and pkgsBuildBuild.stdenv.cc should also end up as the same compiler as pkgsBuildBuild.buildPackages == pkgsBuildBuild.

All that said, changing the dependency from the compiler to all of stdenv is a bit surprising as we don't want any of the rest of the standard build tools, just the compiler. Also, does it even work? stdenv provides the build tools by manipulating PATH inside of the setup script. Merely depending on stdenv shouldn't actually do anything useful.

[Artturin]

@lilyball here's the links

make-derivation.nix: lib.warn if drv.__spliced missing

bash: avoid using unspliced dependency

[lilyball]

The splicing thing is a complete red herring. Using depsBuildBuild = [ buildPackages.stdenv.cc ] gets the correct package, and any equivalent that gets the same package via splicing will have the same problem.