Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't build linux-headers on armv6l #107386

Closed
Gaelan opened this issue Dec 22, 2020 · 17 comments · Fixed by #127875
Closed

Can't build linux-headers on armv6l #107386

Gaelan opened this issue Dec 22, 2020 · 17 comments · Fixed by #127875
Labels
0.kind: bug Something is broken

Comments

@Gaelan
Copy link
Contributor

Gaelan commented Dec 22, 2020

Describe the bug
stdenv.__bootPackages.linuxHeaders (and, therefore, pretty much any package) fails to build on armv6l.

To Reproduce
Steps to reproduce the behavior:

  1. Get an armv6l machine. (I actually used an aarch64 Raspberry Pi 4B.)
  2. cd path/to/nixpkgs; nix-build . --arg system '"armv6l-linux"' -A stdenv.__bootPackages.linuxHeaders
these derivations will be built:
  /nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv
building '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' on 'ssh://[email protected]'...
unpacking sources
unpacking source archive /nix/store/cai15i0m74ihfajdqfz56xgayc9bf9m3-linux-5.9.8.tar.xz
source root is linux-5.9.8
setting SOURCE_DATE_EPOCH to timestamp 1605039377 of file linux-5.9.8/virt/lib/irqbypass.c
patching sources
applying patch /nix/store/69lyjyca86317hdsc1rwf3ahn7s5kiyn-no-relocs.patch
patching file arch/x86/Makefile
Hunk #1 succeeded at 212 (offset -19 lines).
configuring
no configure script, doing nothing
building
  HOSTCC  scripts/basic/fixdep
bash: line 1:  4020 Segmentation fault      (core dumped) scripts/basic/fixdep scripts/basic/.fixdep.d scripts/basic/fixdep 'gcc -Wp,-MMD,scripts/basic/.fixdep.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89         -o scripts/basic/fixdep scripts/basic/fixdep.c   ' > scripts/basic/.fixdep.cmd
make[1]: *** [scripts/Makefile.host:95: scripts/basic/fixdep] Error 139
make[1]: *** Deleting file 'scripts/basic/fixdep'
make: *** [Makefile:539: scripts_basic] Error 2
error: build of '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' on 'ssh://[email protected]' failed: error: --- Error --- nix-daemon
builder for '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' failed with exit code 2; last 10 log lines:
  patching file arch/x86/Makefile
  Hunk #1 succeeded at 212 (offset -19 lines).
  configuring
  no configure script, doing nothing
  building
    HOSTCC  scripts/basic/fixdep
  bash: line 1:  4020 Segmentation fault      (core dumped) scripts/basic/fixdep scripts/basic/.fixdep.d scripts/basic/fixdep 'gcc -Wp,-MMD,scripts/basic/.fixdep.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89         -o scripts/basic/fixdep scripts/basic/fixdep.c   ' > scripts/basic/.fixdep.cmd
  make[1]: *** [scripts/Makefile.host:95: scripts/basic/fixdep] Error 139
  make[1]: *** Deleting file 'scripts/basic/fixdep'
  make: *** [Makefile:539: scripts_basic] Error 2
builder for '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' failed with exit code 1
error: build of '/nix/store/6s0a9wyc446qpsn2svia7c9sc91rxbsr-linux-headers-5.9.8.drv' failed

Additional context

The segfault comes from fixdep, which is invoked by the Linux kernel build system.

Here's a minimal C program that segfaults, when compiled with the cc from nix-shell . --arg system '"armv6l-linux"' -A stdenv.__bootPackages.linuxHeaders:

#include <stdio.h>
int main(int argc, char** argv)
{
	fprintf(stderr, "Usage: fixdep <depfile> <target> <cmdline>\n");
	return 0;
}

A git bisect reveals that the faulty commit is b3640e0 (#103183).

Notify maintainers

@Ericson2314, maintainer of binutils (since that's what seems to be the culprit here)
@expipiplus1, author of #103183

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

  • system: "aarch64-linux"
  • host os: Linux 4.19.118, NixOS, 21.03.20201220.9cb4d2f (Okapi)
  • multi-user?: yes
  • sandbox: yes
  • version: nix-env (Nix) 2.4pre20201201_5a6ddb3
  • channels(gaelan): "nixos-20.09pre225673.8ba41a1e149"
  • channels(root): "nixos-20.09pre224185.ed755a65570"
  • nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:
@Gaelan Gaelan added the 0.kind: bug Something is broken label Dec 22, 2020
@expipiplus1
Copy link
Contributor

I guess that this patch should only be enabled on armv7l then. Truthfully that PR was just changes proposed/written by others and someone else is probably more qualified wrt binutils to say what the correct thing to do would be.

@Gaelan
Copy link
Contributor Author

Gaelan commented Dec 28, 2020

I don't think it's an armv6l vs. armv7l thing: nix-build . --arg system '"armv7l-linux"' -A stdenv.__bootPackages.linuxHeaders fails in the same way.

There's a possibility that the issue is that I'm trying to build on a Pi 4, which is actually a 64-bit machine, with a 64-bit kernel. To test this, I copied the linux-headers closure, and all of its deps, to a Pi 0 W (which is actually armv6l), and got the same bug. It is still possible that the issue arises due to the Pi 4 miscompiling one of the dependences. I'm doing a full build on the Pi 0 to rule this out, but of course that could take a while.

@Gaelan
Copy link
Contributor Author

Gaelan commented Dec 28, 2020

Alright, nix-build . --arg system '"armv6l-linux"' -A stdenv.__bootPackages.linuxHeaders also fails when building from scratch on a Pi 0. At this point, I'm pretty sure this patch just breaks building anything natively on aarch32.

@expipiplus1
Copy link
Contributor

Thanks for the testing @Gaelan. I'm not sure what's more important, cross compiling to arm32 or native compiling! Hopefully @Ericson2314 can shed some light here.

@Ericson2314
Copy link
Member

Ericson2314 commented Jan 3, 2021

What a mess! I guess the patch fixes some relocations breaks others. If we could make minimal programs for both that would be nice, but that's a lot of work.

@Ericson2314
Copy link
Member

Ah I see @expipiplus1 already minimized in https://sourceware.org/bugzilla/show_bug.cgi?id=16177#c9, and we can cross-reference Jessica's patch on this issue too. So maybe once we minimize this issue we'll have enough combinations to try that it shouldn't be so bad.

@tgunnoe
Copy link

tgunnoe commented Jan 8, 2021

I have this problem on armv7l as well, on the novena board. 5.9.8 headers required for building nix fail.

@stefandeml
Copy link

stefandeml commented Jun 15, 2021

Thanks for the testing @Gaelan. I'm not sure what's more important, cross compiling to arm32 or native compiling! Hopefully @Ericson2314 can shed some light here.

I'm running into the exact same issue using arm emulation via qemu-arm-static on my x64 machine.

@Atemu
Copy link
Member

Atemu commented Jun 16, 2021

IIRC GHC was the only package affected by the bug in my NixOS closure when I had it running on an armv7 machine back in 2019 and it was also happening on native builds; had nothing to do with musl or cross.

Couldn't we just make a special binutils with the patch for GHC and the (at worst) handful of other programs which need it?

@stefandeml
Copy link

stefandeml commented Jul 5, 2021

Hello everyone,
I'm trying to make use of the recent changes.

Here what I've done:
I'm using the nix installer from here. I guess this should actually not be needed any more as we now have hydra jobs for this merged 🎉.

Using my Pi4 with 32bit kernel:
pi@raspberrypi:~ $ uname -a Linux raspberrypi 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l GNU/Linux
on the unstable channel (should not be relevant) and on current master running:
cd path/to/nixpkgs; nix-build . --arg system '"armv7l-linux"' -A stdenv.__bootPackages.linuxHeaders
now passes. W-h-o-o 🎉.

However, compiling our friendly greeting:
cd path/to/nixpkgs; nix-build . --arg system '"armv7l-linux"' -A hello
results in the following segfault:

(cd /build/build/elf/.; /nix/store/1ikhs23qaiw1zq793iac3s1m4307dwvk-binutils-2.35.1/bin/objdump -h dl-load.o dl-lookup.o dl-object.o dl-reloc.o dl-deps.o dl-hwcaps.o dl-runtime.o dl-init.o dl-fini.o dl-debug.o dl-misc.o dl-version.o dl-profile.o dl-tls.o dl-origin.o dl-scope.o dl-execstack.o dl-open.o dl-close.o dl-trampoline.o dl-exception.o dl-sort-maps.o dl-lookup-direct.o dl-call-libc-early-init.o dl-cache.o dl-tunables.o tlsdesc.o dl-tlsdesc.o dl-support.o dl-iteratephdr.o dl-addr.o dl-addr-obj.o enbl-secure.o dl-profstub.o dl-libc.o dl-sym.o dl-sysdep.o dl-error.o dl-reloc-static-pie.o libc_early_init.o find_exidx.o dl-iteratephdr.os dl-addr.os dl-addr-obj.os dl-profstub.os dl-libc.os dl-sym.os dl-error.os libc_early_init.os aeabi_unwind_cpp_pr1.os find_exidx.os rtld.os dl-load.os dl-lookup.os dl-object.os dl-reloc.os dl-deps.os dl-hwcaps.os dl-runtime.os dl-init.os dl-fini.os dl-debug.os dl-misc.os dl-version.os dl-profile.os dl-tls.os dl-origin.os dl-scope.os dl-execstack.os dl-open.os dl-close.os dl-trampoline.os dl-exception.os dl-sort-maps.os dl-lookup-direct.os dl-call-libc-early-init.os dl-cache.os dl-tunables.os tlsdesc.os dl-tlsdesc.os dl-sysdep.os dl-environ.os dl-minimal.os dl-error-minimal.os dl-conflict.os aeabi_read_tp.os libc-do-syscall.os dl-brk.os dl-sbrk.os dl-getcwd.os dl-openat64.os dl-opendir.os dl-fxstatat64.os sofini.os interp.os static-stubs.o cache.o readlib.o xmalloc.o xstrdup.o chroot_canon.o sotruss-lib.os sotruss-lib.so) | \
gawk '/\.gnu\.glibc-stub\./ { \
          sub(/\.gnu\.glibc-stub\./, "", $2); \
          stubs[$2] = 1; } \
        END { for (s in stubs) print "#define __stub_" s }' > /build/build/elf/stubsT
mv -f /build/build/elf/stubsT /build/build/elf/stubs
make[2]: Leaving directory '/build/glibc-2.32/elf'
(sed '/^@/d' include/stubs-prologue.h; LC_ALL=C sort /build/build/csu/stubs /build/build/iconv/stubs /build/build/locale/stubs /build/build/localedata/stubs /build/build/iconvdata/stubs /build/build/assert/stubs /build/build/ctype/stubs /build/build/intl/stubs /build/build/catgets/stubs /build/build/math/stubs /build/build/setjmp/stubs /build/build/signal/stubs /build/build/stdlib/stubs /build/build/stdio-common/stubs /build/build/libio/stubs /build/build/dlfcn/stubs /build/build/nptl/stubs /build/build/malloc/stubs /build/build/string/stubs /build/build/wcsmbs/stubs /build/build/timezone/stubs /build/build/time/stubs /build/build/dirent/stubs /build/build/grp/stubs /build/build/pwd/stubs /build/build/posix/stubs /build/build/io/stubs /build/build/termios/stubs /build/build/resource/stubs /build/build/misc/stubs /build/build/socket/stubs /build/build/sysvipc/stubs /build/build/gmon/stubs /build/build/gnulib/stubs /build/build/wctype/stubs /build/build/manual/stubs /build/build/shadow/stubs /build/build/gshadow/stubs /build/build/po/stubs /build/build/argp/stubs /build/build/rt/stubs /build/build/conform/stubs /build/build/debug/stubs /build/build/mathvec/stubs /build/build/support/stubs /build/build/crypt/stubs /build/build/nptl_db/stubs /build/build/inet/stubs /build/build/resolv/stubs /build/build/nss/stubs /build/build/hesiod/stubs /build/build/sunrpc/stubs /build/build/nis/stubs /build/build/nscd/stubs /build/build/login/stubs /build/build/elf/stubs) > /build/build/stubs.h
if test -r /nix/store/ail4z80i8dfd17h35r00rvqgshgaz43f-glibc-2.32-48-dev/include/gnu/stubs-hard.h && cmp -s /build/build/stubs.h /nix/store/ail4z80i8dfd17h35r00rvqgshgaz43f-glibc-2.32-48-dev/include/gnu/stubs-hard.h; \
then echo 'stubs.h unchanged'; \
else /nix/store/cvzil995dqk628n017d02s1zaa6fjh4i-bootstrap-tools/bin/install -c -m 644 /build/build/stubs.h /nix/store/ail4z80i8dfd17h35r00rvqgshgaz43f-glibc-2.32-48-dev/include/gnu/stubs-hard.h; fi
rm -f /build/build/stubs.h
/build/build/elf/sln /build/build/elf/symlink.list
make[1]: *** [Makefile:115: install-symbolic-link] Segmentation fault
make[1]: Leaving directory '/build/glibc-2.32'
make: *** [Makefile:12: install] Error 2
builder for '/nix/store/hrrfkbfywf8awmj84gkasj3zck5p4s3l-glibc-2.32-48.drv' failed with exit code 2
cannot build derivation '/nix/store/f3a805kyq0aa6yix4r5a2rh9c6xli2wz-bootstrap-stage3-gcc-wrapper-.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ivlrj5yxxx9a7z7ixs8faq1gszs4iydx-bootstrap-stage4-gcc-wrapper-10.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/v1gijiqkgzwkr7n534mypjfdrg2sj951-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/nzkrvarvqqnm5wqwdxyj3y4vplcs76dw-hello-2.10.drv': 1 dependencies couldn't be built
error: build of '/nix/store/nzkrvarvqqnm5wqwdxyj3y4vplcs76dw-hello-2.10.drv' failed

Seems like this PR has not been all to fix the broken armv7l target. Anyone aware of the issue here and how to continue?

Happy to update the official Wiki as soon as I've things working ... also including the hydra tarball.
Thanks!

@misuzu
Copy link
Contributor

misuzu commented Jul 6, 2021

@stefandeml try cherry-picking commit from #129004

@nixos-discourse
Copy link

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/new-aarch64-and-armv7l-linux-omg-builders/1010/8

@stefandeml
Copy link

stefandeml commented Jul 6, 2021

@stefandeml try cherry-picking commit from #129004

Thanks 🎉. That definitely helped but failing at stage4 now:

g++ -std=gnu++98 -fno-PIE -c   -g -DIN_GCC     -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-format -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common  -DHAVE_CONFIG_H -I. -I. -I../../gcc-10.3.0/gcc -I../../gcc-10.3.0/gcc/. -I../../gcc-10.3.0/gcc/../include -I../../gcc-10.3.0/gcc/../libcpp/include -I/nix/store/5ydj93x2ih7fwfaakicxgx6n1l3gbwiv-gmp-6.2.1-dev/include -I/nix/store/68jvd28xs49xcajivfnb9dn823v4wxd5-mpfr-4.1.0-dev/include -I/nix/store/4v70af3qw8ryrlghjg9pf0z4j554r28w-libmpc-1.2.1/include  -I../../gcc-10.3.0/gcc/../libdecnumber -I../../gcc-10.3.0/gcc/../libdecnumber/dpd -I../libdecnumber -I../../gcc-10.3.0/gcc/../libbacktrace -I/nix/store/j8fpc8g0sbjpsyzrwrbvb7zn84n72pij-isl-0.20/include  -o insn-extract.o -MT insn-extract.o -MMD -MP -MF ./.deps/insn-extract.TPo insn-extract.c
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[3]: *** [Makefile:1117: insn-emit.o] Error 1
make[3]: *** Waiting for unfinished jobs....
g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[3]: *** [Makefile:1117: gimple-match.o] Error 1
rm gcc.pod
make[3]: Leaving directory '/build/build/gcc'
make[2]: *** [Makefile:4741: all-stage1-gcc] Error 2
make[2]: Leaving directory '/build/build'
make[1]: *** [Makefile:22554: stage1-bubble] Error 2
make[1]: Leaving directory '/build/build'
make: *** [Makefile:22875: bootstrap] Error 2
builder for '/nix/store/16yvw06i2ffynql1g28nkns6hkjr7w1b-gcc-10.3.0.drv' failed with exit code 2
cannot build derivation '/nix/store/x5492ifnq2w4smrdspvdxbw20z8js1hw-bootstrap-stage4-gcc-wrapper-10.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/m4gb0xhc9w847j5liga7fw2hcqwadmkm-stdenv-linux.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/62ck69fyz7gi6jbhicizwvjwjp1gqkwi-hello-2.10.drv': 1 dependencies couldn't be built
error: build of '/nix/store/62ck69fyz7gi6jbhicizwvjwjp1gqkwi-hello-2.10.drv' failed

Hmm, looks like the OS killed it? - maybe OOM?
I guess there is a way to reduce memory load by reducing the number of parallel jobs? Do we know the minimal config right now?

@misuzu
Copy link
Contributor

misuzu commented Jul 6, 2021

Hmm, looks like the OS killed it? - maybe OOM?
I guess there is a way to reduce memory load by reducing the number of parallel jobs?

I had no issues building the entire system (with a few tweaks) on armv7l virtual machine with 4 cores and 4G of RAM.
You can try building with -j1 --cores 1. Do you use NixOS? If not, is nix installed in multi-user mode?

@vcunat
Copy link
Member

vcunat commented Jul 9, 2021

Yes, OOM often produces exactly this "killed" log line.

@Atemu
Copy link
Member

Atemu commented Jul 9, 2021

Also make sure you have zram swap (perhaps even with zstd) if you're running into OOM issues. It's free additional RAM space.

@yu-re-ka
Copy link
Contributor

yu-re-ka commented Oct 3, 2022

Hitting the exact same bug when cross-compiling to armv7l now, in packages that have PIC disabled (zfs)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.kind: bug Something is broken
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants