Bootstrappable Builds #8929
Open
Bootstrappable Builds #8929
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tobtoht
force-pushed
the
guix
branch
2 times, most recently
from
09e44ca
to
ee3964a
Compare
|
I cannot wait until we finalize and merge this. Thanks for your great work @tobtoht |
tobtoht
force-pushed
the
guix
branch
5 times, most recently
from
7ff7aca
to
008cf5e
Compare
tobtoht
force-pushed
the
guix
branch
3 times, most recently
from
a8ea3d2
to
bf1b97d
Compare
|
I don't expect to make major changes to this PR, it's ready for testing. Reviews on any of the sub-PRs would help move this forward. |
tobtoht
commented
Feb 25, 2024
| #!/bin/bash | ||
| if [ "$1" != "-cc1" ]; then | ||
| - `dirname $0`/clang{version} {flags} "$@" | ||
| + env -u C_INCLUDE_PATH -u CPLUS_INCLUDE_PATH -u OBJC_INCLUDE_PATH -u OBJCPLUS_INCLUDE_PATH -u CPATH -u LIBRARY_PATH `dirname $0`/clang{version} {flags} "$@" |
There was a problem hiding this comment.
We need to unset these environment variables here because they refer to Guix profile packages. We want clang to only include headers from the NDK. We can't unset them in build.sh because native_ package builds would fail to find the necessary include headers. Unsetting these variables does not affect non-Guix builds.
There was a problem hiding this comment.
This patch will be removed in #9456
Comment on lines
+11
to
+14
| + #if $(real-version) < "4.0.0" | ||
| + #{ | ||
| + # flags darwin.compile.c++ OPTIONS $(condition) : -fcoalesce-templates ; | ||
| + #} |
There was a problem hiding this comment.
Boost 1.64.0 doesn't recognize that we're building with Clang and passes a flags that results in an error. We don't support GCC < 4.0 at all, so commenting out the lines here is fine. Patch can be dropped when we update Boost.
There was a problem hiding this comment.
This patch will be removed in #9162
| @@ -0,0 +1,76 @@ | |||
| Note that this has been modified from the original commit, to use __has_include | |||
There was a problem hiding this comment.
riscv64-linux-gnu builds fail without this patch
There was a problem hiding this comment.
Is this the same issue with buggy gcc version 10-11 compiler? (Is this solved on gcc <=9 and >=12+?)
There was a problem hiding this comment.
Unrelated, patch fixes a missing include in glibc. Monero codebase doesn't contain __has_include__. Log for reference:
../sysdeps/unix/sysv/linux/riscv/flush-icache.c:24:10: fatal error: asm/syscalls.h: No such file or directory
24 | #include <asm/syscalls.h>
| ^~~~~~~~~~~~~~~~
compilation terminated.
Patch can be removed when we upgrade glibc to > 2.27.
| @@ -0,0 +1,22 @@ | |||
| Without ffile-prefix-map, the debug symbols will contain paths for the | |||
There was a problem hiding this comment.
Building on aarch64 should produce bit-for-bit identical release binaries.
There was a problem hiding this comment.
It now does for linux, windows, and macos targets. FreeBSD and Android targets use pre-compiled x86_64 toolchains and libraries.
| contrib/gitian/docker/ | ||
| contrib/gitian/sigs/ | ||
| # guix | ||
| /guix |
There was a problem hiding this comment.
Finished builds are available in ./guix/guix-build-<COMMIT>/
├── guix-build-222ea1a30058
│ ├── build
│ │ └── distsrc-222ea1a30058-x86_64-linux-gnu # build directory
│ ├── logs
│ │ └── x86_64-linux-gnu # various logs for reproducibility debugging
│ │ ├── depends-gen_id-build.txt
│ │ ├── depends-gen_id-host.txt
│ │ ├── depends-hashes.txt
│ │ ├── depends-packages.txt
│ │ ├── guix-env.txt
│ │ ├── guix-hashes.txt
│ │ └── SHA256SUMS.part
│ ├── output
│ │ └── x86_64-linux-gnu
│ │ └── monero-x86_64-linux-gnu-222ea1a30058.tar.bz2 # binary tarball
│ └── var
│ ├── precious_dirs # directories to keep when running `./contrib/guix/guix-clean`
│ └── profiles
| @@ -267,4 +275,4 @@ $(foreach package,$(all_packages),$(eval $(call int_config_attach_build_config,$ | |||
| $(foreach package,$(all_packages),$(eval $(call int_add_cmds,$(package)))) | |||
|
|
|||
| #special exception: if a toolchain package exists, all non-native packages depend on it | |||
| $(foreach package,$(packages),$(eval $($(package)_unpacked): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) )) | |||
| $(foreach package,$(packages),$(eval $($(package)_extracted): |$($($(host_arch)_$(host_os)_native_toolchain)_cached) )) | |||
There was a problem hiding this comment.
_unpacked target doesn't exist.
tobtoht
commented
Feb 25, 2024
Comment on lines
+3
to
+22
| $(1)_cc=$$($$($(1)_type)_CC) | ||
| $(1)_cxx=$$($$($(1)_type)_CXX) | ||
| $(1)_objc=$$($$($(1)_type)_OBJC) | ||
| $(1)_objcxx=$$($$($(1)_type)_OBJCXX) | ||
| $(1)_ar=$$($$($(1)_type)_AR) | ||
| $(1)_ranlib=$$($$($(1)_type)_RANLIB) | ||
| $(1)_libtool=$$($$($(1)_type)_LIBTOOL) | ||
| $(1)_nm=$$($$($(1)_type)_NM) | ||
| $(1)_cflags=$$($$($(1)_type)_CFLAGS) \ | ||
| $$($$($(1)_type)_$$(release_type)_CFLAGS) | ||
| $(1)_cxxflags=$$($$($(1)_type)_CXXFLAGS) \ | ||
| $$($$($(1)_type)_$$(release_type)_CXXFLAGS) | ||
| $(1)_arflags=$$($$($(1)_type)_ARFLAGS) \ | ||
| $$($$($(1)_type)_$(release_type)_ARFLAGS) | ||
| $(1)_ldflags=$$($$($(1)_type)_LDFLAGS) \ | ||
| $$($$($(1)_type)_$$(release_type)_LDFLAGS) \ | ||
| -L$$($($(1)_type)_prefix)/lib | ||
| $(1)_cppflags=$$($$($(1)_type)_CPPFLAGS) \ | ||
| $$($$($(1)_type)_$$(release_type)_CPPFLAGS) \ | ||
| -I$$($$($(1)_type)_prefix)/include |
There was a problem hiding this comment.
Delay expansion of package variables.
.github/workflows/guix.yml
Outdated
| @@ -0,0 +1,63 @@ | |||
| name: ci/gh-actions/guix | |||
|
|
|||
| on: [push, pull_request] | |||
There was a problem hiding this comment.
I removed guix and depends package caching from the CI workflow because it uses too much space and causes evictions for other caches (10 GB limit). We only need to run it when there is a change to contrib/{depends,guix}, which happens infrequently and doesn't need to complete quickly.
on: [push, pull_request] is temporary for testing.
.github/workflows/guix.yml
Outdated
| path: contrib/depends/sources | ||
| key: sources-${{ hashFiles('contrib/depends/packages/*') }} | ||
| - name: install dependencies | ||
| run: sudo apt update; sudo apt -y install guix git ca-certificates |
There was a problem hiding this comment.
It may be worth investigating if using the installer script is faster.
| (("-rpath=") "-rpath-link=")) | ||
| #t)))))))) | ||
|
|
||
| (define-public glibc-2.27 |
There was a problem hiding this comment.
The minimum glibc version for all Linux targets is now 2.27 (Ubuntu 18.04, Debian 10).
For more context, see: #9171 (comment)
tobtoht
force-pushed
the
guix
branch
6 times, most recently
from
817fcae
to
9ea73c0
Compare
tobtoht
commented
Feb 26, 2024
Comment on lines
-81
to
+84
| SET(CMAKE_C_COMPILER @prefix@/native/bin/clang) | ||
| SET(CMAKE_C_COMPILER @CC@) | ||
| SET(CMAKE_C_COMPILER_TARGET ${CLANG_TARGET}) | ||
| SET(CMAKE_C_FLAGS_INIT -B${_CMAKE_TOOLCHAIN_PREFIX}) | ||
| SET(CMAKE_CXX_COMPILER @prefix@/native/bin/clang++ -stdlib=libc++) | ||
| SET(CMAKE_CXX_COMPILER @CXX@ -stdlib=libc++) |
There was a problem hiding this comment.
We're now using the system / guix-provided clang for darwin builds.
Comment on lines
+87
to
+88
| SET(CMAKE_ASM_COMPILER clang) | ||
| SET(CMAKE_ASM-ATT_COMPILER as) |
There was a problem hiding this comment.
CMake automatically reduces CMAKE_CXX_COMPILER to the first command to derive CMAKE_ASM_COMPILER. This happens to be env instead of clang, so we need to set it explicitly.
Comment on lines
-49
to
+57
| final_build_id_long+=$($(package)_build_id_long) | ||
| final_build_id_long+=:[recipe]:$(1)-$($(1)_version)-$($(1)_recipe_hash)-$(release_type):[deps]$(foreach dep,$($(1)_build_id_deps),$(shell echo ":$(dep)")):[$($(1)_type)_id]:$($($(1)_type)_id_string): |
There was a problem hiding this comment.
Increase verbosity of final_build_id_long to ease debugging reproducibility issues.
Comment on lines
+103
to
+104
| build_id_string:=$(realpath $(GUIX_ENVIRONMENT)) | ||
| $(host_arch)_$(host_os)_id_string:=$(realpath $(GUIX_ENVIRONMENT)) |
There was a problem hiding this comment.
Any change to manifest.scm invalidates the cache.
tobtoht
force-pushed
the
guix
branch
2 times, most recently
from
b824f18
to
dfd23c0
Compare
0xFFFC0000
reviewed
Aug 11, 2024
| fetch-depth: 0 | ||
| submodules: recursive | ||
| - name: remove bundled packages | ||
| run: sudo rm -rf /usr/local |
There was a problem hiding this comment.
Quick question. Do we need this if we are building with GUIX? Since the premise of GUIX is sandboxed builds.
There was a problem hiding this comment.
It's just to free up disk space. See monero-project/monero-gui#4223
nahuhh
reviewed
Aug 11, 2024
|
|
||
| jobs: | ||
| cache-sources: | ||
| runs-on: ubuntu-24.04 |
There was a problem hiding this comment.
Oh, that might fix the AppArmor issue. Good suggestion, will try.
Nvm, GitHub only has Linux runners for Ubuntu.
tobtoht
force-pushed
the
guix
branch
2 times, most recently
from
a3d0081
to
ee3782d
Compare
Matching hashes for Linux, Windows, and macOS targets on x86_64 and aarch64 based machines. |
|
|
Quite a few of the inline comments on github would perhaps better aid future maintainers as source code comments (if it's non-trivial enough to comment, why confine the comments to github, rather than the actual source?) |
This was referenced Aug 25, 2024
|
Guix I have also added comments to the source where useful, as was suggested by @iamamyth. Build failures:
Edit: I'd rather not patch clang as that would add a lot of extra build time for builders that use substitutes and updating clang is too involved for this PR, so I'm reverting the time-machine bump and rolling it into a separate PR. |
Matches GitHub CI. |
|
This is what I got: |
Should I try to find the issue, simply run again or wait? Do you need some additional input besides (k)ubuntu 24.04 based on debian version trixie/sid? |
|
@DiosDelRayo Oh, that's annoying. It's this Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/guix/+bug/2064115 Let me check if there is a reliable workaround / fix. Edit: the solution here works: https://bugs.launchpad.net/ubuntu/+source/guix/+bug/2064115/comments/2 # in a root shell
$ apt install apparmor-utils
$ cat <<EOL >> /etc/apparmor.d/guix
abi <abi/4.0>,
include <tunables/global>
profile guix /usr/bin/guix flags=(unconfined) {
userns,
include if exists <local/guix>
}
EOL
$ /etc/init.d/apparmor reload
$ aa-enforce guix |
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR proposes to replace Gitian with Guix to achieve bootstrappable builds for release binaries.
What?
https://youtube.com/watch?v=I2iShmUTEl8
If you have 15 minutes, please consider watching this presentation by Carl Dong linked above as it covers the migration (Gitian → Guix) this PR proposes to make. It explains why reproducibility alone is not enough.
Other resources:
Why?
Greater build system security: Bootstrappability allows us to audit and reproduce our toolchain instead of blindly trusting binary downloads. Our build environment can be built from source, all the way down. It allows us to reduce our supply chain attack surface by only including the packages that we need, and nothing else.
Easier to set up: Guix runs on any Linux distribution and on most architectures (x86_64, aarch64, riscv64). To produce reproducible release binaries, you only need to install Guix and run the build script. This hopefully leads to more participants in the pre-release verified reproduction process.
More flexibility: Unlike Gitian, we are not limited to the package set of a particular Ubuntu version. Guix allows us to pick and choose our toolchains. This allows us to use the latest compilers while targeting older versions of glibc and would, for example, make it easier to add a modern Rust compiler to our build environment. Packages that are not available in Guix can easily be defined in the manifest or upstreamed.
Better developer UX: Guix allows us to modify any detail about our build environment with ease. Debugging build issues takes less time because we have shell access to the build environment. The monero source code is bind mounted into the container, so edits to package definitions can be tested incrementally.
Future proof: Guix is actively developed, Gitian is in maintenance mode (serious bug fixes only).
How to test?
Requirements:
For more information, see README.md
Notes
53396a22af(28 Aug 2024)* this does not affect run-time kernel requirements, see glibc docs.
The
guix.ymlworkflow does not use caching because it uses too much space and causes evictions for other caches (10 GB limit). We only need to run it when there is a change tocontrib/{depends,guix}, which happens infrequently and doesn't need to complete quickly.This PR removes
native_clangbecause Guix provides us with a bootstrapped package and non-guix builds can use the system clang. This PR introduces backwards incompatible compiler flags, bumping the minimum Clang version to 10 (up from 9) for depends builds. Clang >=10 is available in the package managers of all supported distros (including Ubuntu 18.04).The minimum glibc version for all Linux targets is now 2.27 (Ubuntu 18.04, Debian 10), for more details see #9171 (comment).
The source archive now includes all submodules.
To-do
monero-project/guixrepo--no-substitutesbuildmonero-project/guix.sigsrepoQ&A
What role does Guix serve in the build process?
Guix is used to set-up a reproducible, bootstrappable, containerized build environment. This replaces the Ubuntu 18.04 based environment we used for Gitian builds.
Monero dependencies are built using the
dependsbuild system inside the container provided by Guix. While Guix could eventually replacedepends, it would be too drastic for this PR.Can Guix run on any Linux distribution?
Yes, with few exceptions. Installing Guix on an immutable distro like Fedora SilverBlue might not be possible.
To install Guix, use the official install script: https://guix.gnu.org/manual/en/html_node/Binary-Installation.html
Can Guix run on macOS or Windows?
Not natively. You can run Guix builds on macOS inside a virtual machine (even on Apple Silicon). On Windows, it should be possible to install Guix on WSL2.
Can Guix run on aarch64 / riscv64 machines?
Yes. It even produces the same bit-identical release binaries as Guix run on a x86_64 machine. This isn't possible with Gitian.
Targets that don't currently build on non-x86_64: Android, FreeBSD.
Are Guix builds slower than Gitian?
Guix builds are only slower the first time, because it needs to build toolchains (or the entire package graph if
--no-substitutesis used) from source. Subsequent runs just as fast (or faster) than Gitian due to caching.Why Guix instead of Nix?
Nix packages are not bootstrappable.
What does bootstrappable mean in this context?
It means that the package graph for all packages included in our build environment is rooted in a single 357-byte heavily annotated program. We no longer have to trust binaries (downloaded from Ubuntu servers) for reproducible builds.
For more information: https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/
If a new version of Guix is released, does that mean our build environment is no longer reproducible?
No, builds remain reproducible indefinitely, assuming the source code for all packages is archived and remains available. The exact version of Guix, including all of its packages is pinned using time-machine.
More information:
How do we define which packages are included in the build environment?
In
manifest.scm. In this file, we can choose packages that are already available in Guix or define packages ourselves.When should we define packages in the manifest?
If a package, specific package version, or toolchain isn't available in Guix and it isn't possible to upstream or we can't we can't wait on that.
Is the build environment identical for all targets?
No, extra packages can be added for each target.
If we change the time-machine commit, does that mean our build environment changes?
Yes. The manifest generally does not specify which packages versions are included in the build environment. If a package was updated in Guix between the old and new commit, we will have the updated version in our build environment. Updating the time-machine can therefore cause breakage and should be done with caution.
How do I get shell access to the build environment for development purposes?
In
contrib/guix/guix-buildreplacebash -c "cd /monero && bash contrib/guix/libexec/build.sh"withbash.Then invoke with:
FORCE_DIRTY_WORKTREE=1 ./contrib/guix/guix-build.Where to start review?
contrib/guix/README.md↓
contrib/guix/guix-build↓
contrib/guix/manifest.scm↓
contrib/guix/libexec/build.shTroubleshooting
mount "none" on "/tmp/guix-directory.": Permission deniedyou need to create an apparmor profile for Guix, see: Bootstrappable Builds #8929 (comment)Status
This PR is ready for testing.
Testing checklist
Follow-up PRs