Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stdenvBootstrapTools: update aarch64 musl #283492

Merged
merged 1 commit into from
Jan 25, 2024

Conversation

alyssais
Copy link
Member

@alyssais alyssais commented Jan 24, 2024

A couple of notes first:

  • I'm not sure why Hydra doesn't show the build products on the build page. I found them by inspecting hydra-build-products in the output. But I don't see the build products on aarch64: update bootstrap-files #151399 's Hydra build either, so I guess this has just been broken forever and this is how everybody does it?
  • AFAICT, this is our first time uploading musl tarballs to tarballs.nixos.org, so I had to choose where to put them. We weren't forward-thinking with including the libc in the names of the Glibc bootstrap tarballs, so I've just stuck a "-musl" suffix on the end, but we can change that if it would be better to use some other naming scheme.

The aarch64 musl bootstrap tools are woefully outdated. Just getting them to the point of being able to build new versions of themselves required a number of hacks that can be reverted once we have new bootstrap tools, and before that it stdenv didn't even build for the preceding three years.

So, following the script established by previous bootstrap tools updates:

Files came from this Hydra build:

https://hydra.nixos.org/build/246470544

…which used nixpkgs revision dd5621d to instantiate:

/nix/store/g480ass2vjmakaq03z7k2j95xnxh206a-stdenv-bootstrap-tools.drv

…and then built:

/nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools

I downloaded these files from Hydra and prefetched them into the Nix store with the following commands:

STOREPATH=95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools
OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
nix --extra-experimental-features nix-command store prefetch-file \
  file://$(nix --extra-experimental-features nix-command store add-file --name bootstrap-tools.tar.xz  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
nix --extra-experimental-features nix-command store prefetch-file --executable \
  file://$(nix --extra-experimental-features nix-command store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)

These commands produced the following output:

warning: you did not specify '--add-root'; the result might be removed by the garbage collector
Downloaded 'file:///nix/store/fm8ys5bb737j82xgyiciyzpcwmzyi9b2-bootstrap-tools.tar.xz' to '/nix/store/jml0gh0q2rnc9sgr87irz0jfbl0fq545-fm8ys5bb737j82xgyiciyzpcwmzyi9b2-bootstrap-tools.tar.xz' (hash 'sha256-ZY9IMOmx1VOn6uoFDpdJbTnPX59TEkrVCzWNtjQ8/QE=').
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
Downloaded 'file:///nix/store/rn9fx55mw2s0qg90zdjxa9xn11nlmjpg-busybox' to '/nix/store/9qylz8gqll63pprwkwlyfs9g4zilak2m-rn9fx55mw2s0qg90zdjxa9xn11nlmjpg-busybox' (hash 'sha256-WuOaun7U5enbOy8SuuCo6G1fbGwsO16jhy/oM8K0lAs=').

I used the hashes from the output above to create the fetchurl invocation which is part of this commit.

I then started the bootstrap with the following command:

nix --extra-experimental-features nix-command build -L -f . --arg localSystem '(import ./lib).systems.examples.aarch64-multiplatform-musl' hello

As @lovesegfault requested, here are the the sha256sums of all the on-server components for extra verification:

sha256sum /nix/store/${STOREPATH}/on-server/*

…which produced the following output:

658f4830e9b1d553a7eaea050e97496d39cf5f9f53124ad50b358db6343cfd01  /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
20cdfecb084ddb6b6b958f2b78fd2cc1d9641632f81ec7d5a48fae0a963ad0fa  /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/busybox

@github-actions github-actions bot added the 6.topic: stdenv Standard environment label Jan 24, 2024
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Jan 24, 2024
@lovesegfault
Copy link
Member

I think this is as good a time as any to start using full triples for the bootstrap archive names. What do you think of http://tarballs.nixos.org/stdenv/aarch64-unknown-linux-musl/dd5621df6dcb90122b50da5ec31c411a0de3e538/?

@alyssais
Copy link
Member Author

That sounds sensible to me if you're happy with it. I'll update the PR.

The aarch64 musl bootstrap tools are woefully outdated.  Just getting
them to the point of being able to build new versions of themselves
required a number of hacks[1] that can be reverted once we have new
bootstrap tools, and before that it stdenv didn't even build for the
preceding three years.

[1]: NixOS#169764

So, following the script established by previous bootstrap tools
updates:

Files came from this Hydra build:

	https://hydra.nixos.org/build/246470544

…which used nixpkgs revision dd5621d
to instantiate:

	/nix/store/g480ass2vjmakaq03z7k2j95xnxh206a-stdenv-bootstrap-tools.drv

…and then built:

	/nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools

I downloaded these files from Hydra and prefetched them into the Nix
store with the following commands:

	STOREPATH=95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools
	OPTIONS="--option binary-caches https://cache.nixos.org --option trusted-public-keys cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
	nix --extra-experimental-features nix-command store prefetch-file \
	  file://$(nix --extra-experimental-features nix-command store add-file --name bootstrap-tools.tar.xz  $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/bootstrap-tools.tar.xz)
	nix --extra-experimental-features nix-command store prefetch-file --executable \
	  file://$(nix --extra-experimental-features nix-command store add-path --name busybox $(nix-store ${OPTIONS} -r /nix/store/${STOREPATH})/on-server/busybox)

These commands produced the following output:

	warning: you did not specify '--add-root'; the result might be removed by the garbage collector
	Downloaded 'file:///nix/store/fm8ys5bb737j82xgyiciyzpcwmzyi9b2-bootstrap-tools.tar.xz' to '/nix/store/jml0gh0q2rnc9sgr87irz0jfbl0fq545-fm8ys5bb737j82xgyiciyzpcwmzyi9b2-bootstrap-tools.tar.xz' (hash 'sha256-ZY9IMOmx1VOn6uoFDpdJbTnPX59TEkrVCzWNtjQ8/QE=').
	warning: you did not specify '--add-root'; the result might be removed by the garbage collector
	Downloaded 'file:///nix/store/rn9fx55mw2s0qg90zdjxa9xn11nlmjpg-busybox' to '/nix/store/9qylz8gqll63pprwkwlyfs9g4zilak2m-rn9fx55mw2s0qg90zdjxa9xn11nlmjpg-busybox' (hash 'sha256-WuOaun7U5enbOy8SuuCo6G1fbGwsO16jhy/oM8K0lAs=').

I used the hashes from the output above to create the fetchurl
invocation which is part of this commit.

I then started the bootstrap with the following command:

	nix --extra-experimental-features nix-command build -L -f . --arg localSystem '(import ./lib).systems.examples.aarch64-multiplatform-musl' hello

As @lovesegfault requested, here are the the sha256sums of all the
on-server components for extra verification:

	sha256sum /nix/store/${STOREPATH}/on-server/*

…which produced the following output:

	658f4830e9b1d553a7eaea050e97496d39cf5f9f53124ad50b358db6343cfd01  /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
	20cdfecb084ddb6b6b958f2b78fd2cc1d9641632f81ec7d5a48fae0a963ad0fa  /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/busybox
@lovesegfault
Copy link
Member

Uploaded:

$ STOREPATH=95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools

$ sha256sum /nix/store/${STOREPATH}/on-server/*
658f4830e9b1d553a7eaea050e97496d39cf5f9f53124ad50b358db6343cfd01  /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
20cdfecb084ddb6b6b958f2b78fd2cc1d9641632f81ec7d5a48fae0a963ad0fa  /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/busybox

$ aws s3 cp --recursive --acl public-read /nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/ s3://nixpkgs-tarballs/stdenv/aarch64-unknown-linux-musl/dd5621df6dcb90122b50da5ec31c411a0de3e538/
upload: ../../../../nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/busybox to s3://nixpkgs-tarballs/stdenv/aarch64-unknown-linux-musl/dd5621df6dcb90122b50da5ec31c411a0de3e538/busybox
upload: ../../../../nix/store/95lm0y33dayag4542s8bi83s31bw68dr-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz to s3://nixpkgs-tarballs/stdenv/aarch64-unknown-linux-musl/dd5621df6dcb90122b50da5ec31c411a0de3e538/bootstrap-tools.tar.xz

$ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv/aarch64-unknown-linux-musl/dd5621df6dcb90122b50da5ec31c411a0de3e538/ ./
download: s3://nixpkgs-tarballs/stdenv/aarch64-unknown-linux-musl/dd5621df6dcb90122b50da5ec31c411a0de3e538/busybox to ./busybox
download: s3://nixpkgs-tarballs/stdenv/aarch64-unknown-linux-musl/dd5621df6dcb90122b50da5ec31c411a0de3e538/bootstrap-tools.tar.xz to ./bootstrap-tools.tar.xz

$ sha256sum bootstrap-tools.tar.xz busybox
658f4830e9b1d553a7eaea050e97496d39cf5f9f53124ad50b358db6343cfd01  bootstrap-tools.tar.xz
20cdfecb084ddb6b6b958f2b78fd2cc1d9641632f81ec7d5a48fae0a963ad0fa  busybox

@alyssais alyssais merged commit ea67e45 into NixOS:master Jan 25, 2024
19 checks passed
@alyssais alyssais deleted the aarch64-musl-bootstrap branch January 25, 2024 17:14
@alyssais alyssais mentioned this pull request Jan 25, 2024
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: stdenv Standard environment 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants