<nix/fetchurl.nix> auto-uncompresses xz files

[cleverca22]

[clever@system76:~/apps/nix]$ git bisect log
git bisect start
# bad: [ed6c646f44e5b9e0fcbc53058491e97875a263c2] Doh
git bisect bad ed6c646f44e5b9e0fcbc53058491e97875a263c2
# good: [34c17fdae535d0128d178dde676f56f1bc8439e3] Add a test for LegacySSHStore that doesn't require a VM
git bisect good 34c17fdae535d0128d178dde676f56f1bc8439e3
# only skipped commits left to test
# possible first bad commit: [ed6c646f44e5b9e0fcbc53058491e97875a263c2] Doh
# possible first bad commit: [7de3e00ad905bba85abadd86b83973fdba8d0dfd] Fix setting Content-Encoding in S3 uploads
# possible first bad commit: [d3761f5f8bce1e4c8dcfdff3fa77c173157c0346] Fix Brotli decompression in 'nix log'
# possible first bad commit: [fa4def3d4675c8b2d6aacb56959dbbf9e52df66a] Require libbrotli
# possible first bad commit: [4361a4331f1b2eb3fcfd954c96de353c9d516508] Fix reporting of HTTP body size when a result callback is used
# possible first bad commit: [2825e05d21ecabc8b8524836baf0b9b05da993c6] Make adding paths via nix-store --serve run in constant memory

nix-build ~/nixpkgs/pkgs/stdenv/linux/bootstrap-files/x86_64-musl.nix -A bootstrapTools
with older versions of nix, this correctly fetches the .tar.xz and the hash check passes
with master (and the version in nixos-unstable), it fetches the .tar.xz and then uncompresses it, resulting in a /nix/store/hash.tar.xz that contains only a tar, and therefore, fails the hashcheck

it also doesnt fail for any other arch, because the binary cache has a copy of the file, and is masking the failure

[nh2]

@cleverca22 got this when downloading an xz file when running the command on NixOS/nixpkgs#49071 (comment), namely with this file:

https://github.com/NixOS/nixpkgs/blob/92a047a6c4d46a222e9c323ea85882d0a7a13af8/pkgs/stdenv/linux/bootstrap-files/x86_64-musl.nix#L8

(Note that this file isn't to blame, it's Nix's logic that changed and made the sha256 invalid that way.)

[nh2]

CC @dtzWill

[nh2]

From @cleverca22 :

the first good commit is on 2.1.3, as is the first bad commit
so i would expect 2.1.3 and master to be broken

[dtzWill]

I can reproduce this with nixUnstable but not nix, on current nixos-unstable.

[dtzWill]

Commands to reproduce the working and non-working situations, don't run if you put valuable things in ~/test-store (also if you do you're kinda weird but that doesn't mean you deserve your files to be deleted by accident):

Works:

$ chmod u+rw -R ~/test-store && rm -rf ~/test-store; nix run -f channel:nixos-unstable nix -c nix-build '<nixpkgs/pkgs/stdenv/linux/bootstrap-files/x86_64-musl.nix>' -I nixpkgs=channel:nixos-unstable -A bootstrapTools -vvv --tarball-ttl 0 --store $HOME/test-store --substituters "

Fails:

chmod u+rw -R ~/test-store && rm -rf ~/test-store; nix run -f channel:nixos-unstable nixUnstable -c nix-build '<nixpkgs/pkgs/stdenv/linux/bootstrap-files/x86_64-musl.nix>' -I nixpkgs=channel:nixos-unstable -A bootstrapTools -vvv --tarball-ttl 0 --store $HOME/test-store --substituters ""

Contrary (I think?) to what's reported above (or maybe I misread), it seems 2.1.3 works fine but unstable has the problem?

Hmm, just checked and unstable appears to be 2.1 -- that is, older than nix -- which makes sense to be encountering bugs that were fixed in (and the motivation for) subsequent point releases.

[dtzWill]

The commit log for 2.1.3 since that release (2.1..2.1.3) contains the fix titled "fetchurl: Respect unpack" so that explanation seems pretty consistent :).

Good news is this seems to suggest there is no problem lurking.
Bad news is I'm not sure why this isn't what @cleverca22 is encountering, and I'm not sure why that would be. Fingers crossed it's something simple like a typo or so :).

(seems to be dupe of #2393)

[stale]

I marked this as stale due to inactivity. → More info

[stale]

I closed this issue due to inactivity. → More info