cross compiling rustc-1.60.0 fails (build on x86_64, target aarch64, QEMU)

[karthanistyr]

Steps To Reproduce

Trying to cross-compile a full aarch64 system on a x86_64 machine (arch linux set up as per https://nixos.wiki/wiki/NixOS_on_ARM#Compiling_through_QEMU)

Using this flake:

{
  inputs.nixpkgs.url = "nixpkgs/nixos-22.05";
  outputs = { self, nixpkgs }: rec {
    nixosConfigurations.rpi2 = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
        {
          nixpkgs.config.allowUnsupportedSystem = true;
          nixpkgs.crossSystem.system = "aarch64-linux";
          # ... extra configs as above
        }
      ];
    };
    images.rpi2 = nixosConfigurations.rpi2.config.system.build.sdImage;
  };
}

flake.lock

 karthanis@karth-dev-book ~/Documents/nixos/nixos-kiosk (git)-[main] % cat flake.lock
{
  "nodes": {
    "nixpkgs": {
      "locked": {
        "lastModified": 1658985539,
        "narHash": "sha256-aRVZGndeuUct3S3T6vqOO64D9qY1F7qNTljd0zuwzak=",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "174e938d593817f2eb5ae363684dea7c412eb96a",
        "type": "github"
      },
      "original": {
        "id": "nixpkgs",
        "ref": "nixos-22.05",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
}

and then nix build .\#images.rpi2 (backslash is because zsh)

Console output:

karthanis@karth-dev-book ~/Documents/nixos/nixos-kiosk (git)-[main] % nix build .\#images.rpi2
warning: Git tree '/home/karthanis/Documents/nixos/nixos-kiosk' is dirty
trace: warning: system.stateVersion is not set, defaulting to 22.05. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
error: builder for '/nix/store/d0d9yl1p52dljdgd2qh9km01jdj2k8y1-rustc-1.60.0.drv' failed with exit code 2;
       last 10 log lines:
       >   --- stderr
       >
       >
       >   error occurred: Command "/nix/store/inscsxqv77lmn15hf5r4rvhxhfw5hj54-aarch64-unknown-linux-gnu-stage-final-gcc-wrapper-9.3.0/bin/aarch64-unknown-linux-gnu-c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I/nix/store/jaa9y3a04r9vlcs0mpar9fbpvvsmia24-llvm-14.0.1-dev/include" "-std=c++14" "-fno-exceptions" "-D_GNU_SOURCE" "-D__STDC_CONSTANT_MACROS" "-D__STDC_FORMAT_MACROS" "-D__STDC_LIMIT_MACROS" "-DLLVM_COMPONENT_AARCH64" "-DLLVM_COMPONENT_AMDGPU" "-DLLVM_COMPONENT_ARM" "-DLLVM_COMPONENT_ASMPARSER" "-DLLVM_COMPONENT_AVR" "-DLLVM_COMPONENT_BITREADER" "-DLLVM_COMPONENT_BITWRITER" "-DLLVM_COMPONENT_BPF" "-DLLVM_COMPONENT_COVERAGE" "-DLLVM_COMPONENT_HEXAGON" "-DLLVM_COMPONENT_INSTRUMENTATION" "-DLLVM_COMPONENT_IPO" "-DLLVM_COMPONENT_LINKER" "-DLLVM_COMPONENT_LTO" "-DLLVM_COMPONENT_MIPS" "-DLLVM_COMPONENT_MSP430" "-DLLVM_COMPONENT_NVPTX" "-DLLVM_COMPONENT_POWERPC" "-DLLVM_COMPONENT_RISCV" "-DLLVM_COMPONENT_SPARC" "-DLLVM_COMPONENT_SYSTEMZ" "-DLLVM_COMPONENT_WEBASSEMBLY" "-DLLVM_COMPONENT_X86" "-DNDEBUG" "-o" "/build/rustc-1.60.0-src/build/aarch64-unknown-linux-gnu/stage0-rustc/aarch64-unknown-linux-gnu/release/build/rustc_llvm-68341dfed4fda773/out/llvm-wrapper/PassWrapper.o" "-c" "llvm-wrapper/PassWrapper.cpp" with args "aarch64-unknown-linux-gnu-c++" did not execute successfully (status code exit status: 1).
       >
       >
       > warning: build failed, waiting for other jobs to finish...
       > error: build failed
       > Build completed unsuccessfully in 0:32:16
       > make: *** [Makefile:12: all] Error 1
       For full logs, run 'nix log /nix/store/d0d9yl1p52dljdgd2qh9km01jdj2k8y1-rustc-1.60.0.drv'.
error: 1 dependencies of derivation '/nix/store/19h9s4fg9h7904g5vr9gbiq4xab3z3ny-make-initrd-ng-aarch64-unknown-linux-gnu-0.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9iwhpdaf35d2hby76kh2bxpzlfd9mnxk-spidermonkey-aarch64-unknown-linux-gnu-78.15.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/gk7ff76d1hwj0k684k5skwdjm5ppcp94-polkit-aarch64-unknown-linux-gnu-0.120.drv' failed to build
error: 1 dependencies of derivation '/nix/store/nhbbyc17v4rn6fzyfjspbl21dhpyyymz-unit-generate-shutdown-ramfs.service.drv' failed to build
error: 1 dependencies of derivation '/nix/store/wqaw4wpgyvjzcknr5pkkdsvkzdqkk7mc-nixos-system-nixos-22.05.20220728.174e938.drv' failed to build
error: 1 dependencies of derivation '/nix/store/krhqnns0x2kknlmdighnkk3af68l0kr5-ext4-fs.img.zst-aarch64-unknown-linux-gnu.drv' failed to build
error: 1 dependencies of derivation '/nix/store/7m3jh1kl9cv9gh0yy75cjwfsfzg195zq-nixos-sd-image-22.05.20220728.174e938-aarch64-linux.img-aarch64-unknown-linux-gnu.drv' failed to build
nix build .\#images.rpi2  41,38s user 36,70s system 0% cpu 9:05:59,76 total

Build log

Output of nix log /nix/store/d0d9yl1p52dljdgd2qh9km01jdj2k8y1-rustc-1.60.0.drv
Gist: https://gist.github.com/karthanistyr/ae79df64ff783bfee26af0da7c2e7369

Notify maintainers

@madjar @cstrahan @globin @Havvy

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

karthanis@karth-dev-book ~/Documents/nixos/nixos-kiosk (git)-[main] % nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.18.12-arch1-1, Arch Linux, noversion, rolling`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.10.3`
 - channels(karthanis): `"nixpkgs-22.05"`
 - nixpkgs: `/home/karthanis/.nix-defexpr/channels/nixpkgs`

[Mindavi]

Hmm, I don't recall rustc being broken on 22.05. Note that the cross-compilation support in master is generally better than in the release branches.

I'd like to improve that but for now my recommendation is using master (or one of the unstable branches).

[karthanistyr]

Thanks, I'll try now with unstable and then master if it fails again...
Any thought on why the cpp compiler would fail to include from stdlib though ? On a surface level the command looks fine (sets the cpp standard to C++14 where vector requires C++11) but I'm not skilled enough yet with nix to dig deeper

[alyssais]

It seems like you might be confused between two different methods for building images for other platforms.

One is using QEMU to emulate native builds for the platform you're building for, and the other is cross compilation (using compilers etc. native to your build system to produce binaries for the system you're building for).

nixpkgs.crossSystem.system is about the latter (cross compilation), and it's likely that Nixpkgs won't do the right thing if you tell it to cross compile from one platform to that exact same platform (see also #21471).

So try either using QEMU emulation (without nixpkgs.crossSystem.system), or try cross-compiling (without QEMU emulation). I recommend the latter, as any failures you encounter will likely be much less subtle (if something doesn't work, it will fail for reasons it's possible to understand, rather than because QEMU emulated something slightly wrong), and you won't end up with build impurities due to QEMU emulating things slightly differently to Linux on hardware.

[karthanistyr]

thanks a lot for this, i did actually confuse things in my attempt. Removing the QEMU layer allowed things to progress further and while I'm hitting #182287 right now, I guess I can use either unstable or wait for #182521 to make it to 22.05 proper.

[karthanistyr]

For reference, here's flake.nix now

{
  inputs.nixpkgs.url = "nixpkgs/nixos-22.05";
  outputs = { self, nixpkgs }: rec {
    nixosConfigurations.rpi2 = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      modules = [
        "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
        {
          nixpkgs.config.allowUnsupportedSystem = true;
          nixpkgs.crossSystem.system = "aarch64-linux";
          # ... extra configs as above
        }
      ];
    };
    images.rpi2 = nixosConfigurations.rpi2.config.system.build.sdImage;
  };
}

and /etc/nix/nix.conf with qemu emulation disabled:

[...]
#extra-platforms = aarch64-linux arm-linux