hwdb fails to build during nixos-rebuild on 17.09 channel

[Moredread]

Issue description

I currently can't build the current nixos 17.09 channel after and on commit b81861f. That commit belongs to issue #34786.

Error messages at the end of nixos-rebuild:

patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/kmod...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/e2fsck...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/tune2fs...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/btrfs...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/cryptsetup...
testing patched programs...
hello world
/build/.attr-0: line 113: /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/udevadm: cannot execute binary file: Exec format error
builder for '/nix/store/w7pbndgwj6xr3szaifyqq9ym4y5nbjw2-extra-utils.drv' failed with exit code 126
cannot build derivation '/nix/store/99fi6dqvr6lxdvj307kzm7igz55x1j1h-udev-rules.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ykah3cq9fv8h7yxxgzkc4xs8rn09206x-stage-1-init.sh.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/71wc3jpp6h8dm8y3l90888zc3vayn0aq-initrd.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/60zj16rpk4kplijmximncy8rv0f8lnx0-nixos-system-grable-17.09.git.c831224.drv': 2 dependencies couldn't be built
error: build of '/nix/store/60zj16rpk4kplijmximncy8rv0f8lnx0-nixos-system-grable-17.09.git.c831224.drv' failed

Steps to reproduce

I don't have a minimal working example, but my config is available at https://github.com/Moredread/nixos-config with the machine config in machines/grable.nix

Building that on the 17.09 fails.

Technical details

• system: "86_64-linux"
• host os: Linux 4.15.0, NixOS, 17.09.2905.c1d9aff56e0 (Hummingbird)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 1.12pre5873_b76e282d
• channels(root): "nixos-17.09, nixos-unstable-18.03pre126729.2e4aded3669"
• channels(addy): ""
• nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[Moredread]

Actually the error seems to be in building hwdb, below a more complete log. Any idea what could cause this? I don't know how to debug this further.

nixos-rebuild build -I nixpkgs=$HOME/nixpkgs 
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/lfn83fjqz33dlsghbm896crnw1x5z3mi-system-path.drv
  /nix/store/1zpdki13dfbbkpdrhndhxaqvzdya5aqi-dbus-1.drv
  /nix/store/w7pbndgwj6xr3szaifyqq9ym4y5nbjw2-extra-utils.drv
  /nix/store/99fi6dqvr6lxdvj307kzm7igz55x1j1h-udev-rules.drv
  /nix/store/9mzgqznvlfap6g8dj8y3rs759inc4p01-unit-dbus.service.drv
  /nix/store/dnsd0l8g2r4mbapfdz9qjc3xbafi5ig9-user-units.drv
  /nix/store/vmz5srkn3njnk1mhfriqvpxzrvs140k0-stage-1-init.sh.drv
  /nix/store/ljxb0563hqb545gr8pji976lq8khndkj-initrd.drv
  /nix/store/ndyk57524yy7g3mwk8qkby04b69qfzsd-hwdb.bin.drv
  /nix/store/mhsvig8gp5p6bnn4vpr2i91ckba4m562-unit-polkit.service.drv
  /nix/store/w9jxb1pll2ggvh1nhbydivbxxyryk222-unit-systemd-fsck-.service.drv
  /nix/store/psly51kdz2dpyfyanryd3lqyy8hi2xk0-system-units.drv
  /nix/store/wrnpw0rbv6dlwmmqp54yw5m6svxbxw88-etc.drv
  /nix/store/g88f8f2kva1z0ns2zy0sapda29622ycx-nixos-system-grable-17.09.git.c632884.drv
building '/nix/store/ndyk57524yy7g3mwk8qkby04b69qfzsd-hwdb.bin.drv'...
building '/nix/store/lfn83fjqz33dlsghbm896crnw1x5z3mi-system-path.drv'...
building '/nix/store/w7pbndgwj6xr3szaifyqq9ym4y5nbjw2-extra-utils.drv'...
Adding hwdb files for package /nix/store/4lczhf3yxjrgankyky6ydiq7wvwsaax6-systemd-234
Adding hwdb files for package /nix/store/x7jvl69z9kpgm85nzan0fg4hqy5k2r1a-mdadm-3.3.4
Adding hwdb files for package /nix/store/x357m8pa2d5bpmjnnx75wv49ccsa1zkw-lvm2-2.02.173
Adding hwdb files for package /nix/store/q0r6c1qz9kaiccdv8wp00v9mjdr48fh3-bcache-tools-1.0.7
Adding hwdb files for package /nix/store/hmfy6h2p7cfr897ajgp7lklsnpp6q2qr-libinput-1.7.3
Adding hwdb files for package /nix/store/gkwzvpkbqp6yyw49q9cm9fnhbpvrlrg2-ModemManager-1.6.8
Adding hwdb files for package /nix/store/j9vca4r8jg6awdrcw4zcqkgx2qgypcp4-network-manager-1.6.2
Adding hwdb files for package /nix/store/7fz5flqh6b376g22qh8k6dda9mjd4im7-NetworkManager-fortisslvpn-gnome-1.2.4
Adding hwdb files for package /nix/store/hjx6ppgbah408m9lawsl6bz8zmb8cg10-NetworkManager-iodine-gnome-1.2.0
Adding hwdb files for package /nix/store/8c6wqq8vpf2nlkzgmi4cwj7z1jnkms4k-NetworkManager-l2tp-gnome-1.2.4
Adding hwdb files for package /nix/store/nds2b92d6zpcr9b7x33xd3yzb76jkf1h-NetworkManager-openconnect-gnome-1.2.4
Adding hwdb files for package /nix/store/vkz5m9xg405zs6jivsvpkmp5y6pww1lp-NetworkManager-openvpn-gnome-1.2.8
Adding hwdb files for package /nix/store/cx0an1hqzhdlnin427p6rqpvc30frxxx-NetworkManager-pptp-gnome-1.2.4
Adding hwdb files for package /nix/store/pr89nwq3357qdg43x1qdisaagbqygjxa-NetworkManager-vpnc-gnome-1.2.4
Adding hwdb files for package /nix/store/0p6671cr27ajypqglxpwf4qpvayjkhr2-wpa_supplicant-2.6
Adding hwdb files for package /nix/store/33k33h9585mmc84mngigpzj9bal284wj-udisks-2.1.6
Adding hwdb files for package /nix/store/g8l4shn4rc1z274z871s1frnnjf3922m-extra-udev-rules
Adding hwdb files for package /nix/store/djm8dinn35f5rv3ppghg3v32zl3cgszf-extra-hwdb-file
Adding hwdb files for package /nix/store/9djlq5amzns78mcvqqmgpnbz8ncxgg0l-trezord-udev-rules
Adding hwdb files for package /nix/store/20c1b93fr8lhhx93xhkvjpcv0wlqaj6z-tlp-1.0
Adding hwdb files for package /nix/store/xwymb29wimlahscpmwjzfdhpnddgvnjj-sane-backends-2017-12-01
Adding hwdb files for package /nix/store/31nldaiad46r7dsk1girqs2l30zwi8f0-net.conf
Adding hwdb files for package /nix/store/rsa6v1sswphmm5fw978wimfvhngdkk0w-bluez-5.47
Adding hwdb files for package /nix/store/c3r4b69g01s9c09npfd2f2zmhriviz8f-alsa-utils-1.1.4
Generating hwdb database...
builder for '/nix/store/ndyk57524yy7g3mwk8qkby04b69qfzsd-hwdb.bin.drv' failed with exit code 126
'/nix/store/rpwlq65d18jz11iisnfsmagl6z9xq5lc-busybox-1.27.2/sbin/[' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/['
'/nix/store/rpwlq65d18jz11iisnfsmagl6z9xq5lc-busybox-1.27.2/sbin/[[' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/[['
'/nix/store/d54amiggq6bw23jw6mdsgamvs6v1g3bh-glibc-2.25-123/lib/libc-2.25.so' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libc.so.6'
(...)
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/blkid
'/nix/store/7r7gp3jh0h5v46szfpcnlldpcwg750ql-util-linux-2.31/lib/libblkid.so.1.1.0' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libblkid.so.1'
'/nix/store/7r7gp3jh0h5v46szfpcnlldpcwg750ql-util-linux-2.31/lib/libuuid.so.1.3.0' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libuuid.so.1'
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/dmsetup
'/nix/store/x357m8pa2d5bpmjnnx75wv49ccsa1zkw-lvm2-2.02.173/lib/libdevmapper.so.1.02' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libdevmapper.so.1.02'
'/nix/store/d54amiggq6bw23jw6mdsgamvs6v1g3bh-glibc-2.25-123/lib/librt-2.25.so' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/librt.so.1'
'/nix/store/ayhhalil9km4dsisfakybjqxdlnnbbi7-systemd-234-lib/lib/libudev.so.1.6.6' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libudev.so.1'
'/nix/store/d54amiggq6bw23jw6mdsgamvs6v1g3bh-glibc-2.25-123/lib/libpthread-2.25.so' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libpthread.so.0'
'/nix/store/d54amiggq6bw23jw6mdsgamvs6v1g3bh-glibc-2.25-123/lib/libresolv-2.25.so' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libresolv.so.2'
'/nix/store/lk1m2j63fkhxccg1d4p79wqirl9j5816-libcap-2.25-lib/lib/libcap.so.2.25' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libcap.so.2'
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/lvm
'/nix/store/d54amiggq6bw23jw6mdsgamvs6v1g3bh-glibc-2.25-123/lib/libdl-2.25.so' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libdl.so.2'
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/mdadm
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/mdmon
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/systemd-udevd
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/udevadm
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/ata_id
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/cdrom_id
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/scsi_id
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/v4l_id
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/kmod
'/nix/store/y4p3gg22jjr223r2r681nmfdfvj245lk-xz-5.2.3/lib/liblzma.so.5.2.3' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/liblzma.so.5'
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/e2fsck
'/nix/store/cfkw4f7gp22gnvvbdjdriq18l4h5p0v7-e2fsprogs-1.43.4/lib/libext2fs.so.2.4' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libext2fs.so.2'
'/nix/store/cfkw4f7gp22gnvvbdjdriq18l4h5p0v7-e2fsprogs-1.43.4/lib/libcom_err.so.2.1' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libcom_err.so.2'
'/nix/store/cfkw4f7gp22gnvvbdjdriq18l4h5p0v7-e2fsprogs-1.43.4/lib/libe2p.so.2.3' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libe2p.so.2'
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/tune2fs
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/btrfs
'/nix/store/dl66jvc19im9lqslzqszsp2gif2y0y8i-zlib-1.2.11/lib/libz.so.1.2.11' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libz.so.1'
'/nix/store/9injx3rhmp8qv1v4n3r64vpxdd9ynxcf-lzo-2.10/lib/liblzo2.so.2.0.0' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/liblzo2.so.2'
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/cryptsetup
cannot build derivation '/nix/store/wrnpw0rbv6dlwmmqp54yw5m6svxbxw88-etc.drv': 1 dependencies couldn't be built
'/nix/store/0rwvr9i8sw5bapk4xs8frx0gj0g3v9kk-cryptsetup-1.7.5/lib/libcryptsetup.so.4.7.0' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libcryptsetup.so.4'
'/nix/store/q19aia05vb9c9syvhv1hs911nilhf6yd-popt-1.16/lib/libpopt.so.0.0.0' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libpopt.so.0'
'/nix/store/1j85y9zswjnya4lxsw68agdn6nxg907r-openssl-1.0.2n/lib/libssl.so.1.0.0' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libssl.so.1.0.0'
'/nix/store/1j85y9zswjnya4lxsw68agdn6nxg907r-openssl-1.0.2n/lib/libcrypto.so.1.0.0' -> '/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/libcrypto.so.1.0.0'
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/cryptsetup-askpass
Copying libs for executable /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib/ld-linux-x86-64.so.2
stripping (with flags -s) in /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/lib  /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/systemd-udevd...
not an ELF executable
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/udevadm...
not an ELF executable
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/cryptsetup-askpass...
not an ELF executable
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/busybox...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/blkid...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/dmsetup...
warning: working around a Linux kernel bug by creating a hole of 2105344 bytes in ‘/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/dmsetup’
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/lvm...
warning: working around a Linux kernel bug by creating a hole of 4399104 bytes in ‘/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/lvm’
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/mdadm...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/mdmon...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/ata_id...
warning: working around a Linux kernel bug by creating a hole of 4096 bytes in ‘/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/ata_id’
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/cdrom_id...
warning: working around a Linux kernel bug by creating a hole of 4096 bytes in ‘/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/cdrom_id’
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/scsi_id...
warning: working around a Linux kernel bug by creating a hole of 4096 bytes in ‘/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/scsi_id’
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/v4l_id...
warning: working around a Linux kernel bug by creating a hole of 4096 bytes in ‘/nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/v4l_id’
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/kmod...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/e2fsck...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/tune2fs...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/btrfs...
patching /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/cryptsetup...
testing patched programs...
hello world
/build/.attr-0: line 113: /nix/store/ndlwxwyjkcdh5wp9xz59i81plqxkfwn5-extra-utils/bin/udevadm: cannot execute binary file: Exec format error
builder for '/nix/store/w7pbndgwj6xr3szaifyqq9ym4y5nbjw2-extra-utils.drv' failed with exit code 126
cannot build derivation '/nix/store/99fi6dqvr6lxdvj307kzm7igz55x1j1h-udev-rules.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vmz5srkn3njnk1mhfriqvpxzrvs140k0-stage-1-init.sh.drv': 2 dependencies couldn't be built
cannot build derivation '/nix/store/ljxb0563hqb545gr8pji976lq8khndkj-initrd.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/g88f8f2kva1z0ns2zy0sapda29622ycx-nixos-system-grable-17.09.git.c632884.drv': 2 dependencies couldn't be built
error: build of '/nix/store/g88f8f2kva1z0ns2zy0sapda29622ycx-nixos-system-grable-17.09.git.c632884.drv' failed

[vcunat]

That's suspicious, and it doesn't happen for me on that commit. I'm getting a different derivation hash, probably because of some different options, but none of these should result into the error, IMHO. I suspect FS corruption.

In your place I'd try

nix-store --verify-path /nix/store/4lczhf3yxjrgankyky6ydiq7wvwsaax6-systemd-234

or maybe on the whole nix store

nix-store --verify --check-contents

[Moredread]

Yeah, there seems to be some corruption going on, but nix-store --repair-path didn't work until I disabled the nix.autoOptimiseStore option. Maybe it links without checking the content of the .links location?

Now it works though and I hope verifying the whole store works correctly without auto-optimization.

[vcunat]

.links are content-addressed, so I don't think the content should be checked every time. If FS is losing data, it's hard to defend on higher levels.

[Moredread]

A nix-store --verify --repair or nix-store --repair-path though I would expect to fix such issues.

[Moredread]

BTW checking/repairing my store without auto optimization fixed the issue, so I'm closing the issue.

[vcunat]

That auto-optimization is here: NixOS/nix#1281

[Moredread]

@vcunat thanks :)

[gallexme]

same problem here using nixops to deploy my machine:

rning: working around a Linux kernel bug by creating a hole of 2097152 bytes in ‘/nix/store/brxh74wgp086n7y89zznmd1ylkcnsxws-extra-utils/bin/systemd-udevd’
patching /nix/store/brxh74wgp086n7y89zznmd1ylkcnsxws-extra-utils/bin/tune2fs...
patching /nix/store/brxh74wgp086n7y89zznmd1ylkcnsxws-extra-utils/bin/udevadm...
warning: working around a Linux kernel bug by creating a hole of 2097152 bytes in ‘/nix/store/brxh74wgp086n7y89zznmd1ylkcnsxws-extra-utils/bin/udevadm’
patching /nix/store/brxh74wgp086n7y89zznmd1ylkcnsxws-extra-utils/bin/v4l_id...
warning: working around a Linux kernel bug by creating a hole of 2097152 bytes in ‘/nix/store/brxh74wgp086n7y89zznmd1ylkcnsxws-extra-utils/bin/v4l_id’
testing patched programs...
hello world
/tmp/nix-build-extra-utils.drv-0/.attr-0: line 96: /nix/store/brxh74wgp086n7y89zznmd1ylkcnsxws-extra-utils/bin/udevadm: cannot execute binary file: Exec format error