build: Add Rust to reprobuilds so we can bundle the cln-grpc plugin with it #5421
Conversation
Where you able to reproducibly build the test release? |
Damnt! i forgot to submit the new review! really? :/ |
There was a problem hiding this comment.
Mh I just acked a couple hours ago when the build start on my normal computer before finish the space and abort the process in the middle.
From a clean cloud machine I have problems, so I will redo my review and adding the problem that I'm facing below.
In addition, I did not found the v0.99.1 tag in your repo and I just checkout the commit cdecker@f9bcb25
There is anythings that I'm doing wrong?
| @@ -2,6 +2,8 @@ FROM bionic | |||
|
|
|||
| ENV TZ=UTC | |||
| RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone | |||
| ENV RUST_PROFILE=release | |||
There was a problem hiding this comment.
Mh! by following the what there is written in https://lightning.readthedocs.io/REPRODUCIBLE.html#builder-image-setup during the command exectution
sudo docker build -t cl-repro-bionic - < Dockerfile.bionic
i receive the following message
gitpod /workspace/tmp/lightning/contrib/reprobuild ((f9bcb2510...)) $ sudo docker build -t cl-repro-bionic - < Dockerfile.bionic
Sending build context to Docker daemon 3.584kB
Step 1/13 : FROM bionic
pull access denied for bionic, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
assuming that we need to update the build system? and you are not see this because you have the image already cloned?
There was a problem hiding this comment.
You need to also build the base image, a couple of lines above: https://lightning.readthedocs.io/REPRODUCIBLE.html#base-image-creation
That will build the ubuntu bionic image for example using solely the installation medium contents (fetched from the web), and then we build on top of that known good state.
There was a problem hiding this comment.
To get the tags you might have to git fetch --tags, but I thought it should work out of the box. If that doesn't work you can always git fetch cdecker refs/tags/v0.99.1:v0.99.1
There was a problem hiding this comment.
To get the tags you might have to git fetch --tags, but I thought it should work out of the box. If that doesn't work you can always git fetch cdecker refs/tags/v0.99.1:v0.99.1
Mh! I do not find any tag with this name on your repository https://github.com/cdecker/lightning/tags
There was a problem hiding this comment.
Oops, my bad:
git push origin --tags
Enumerating objects: 18, done.
Counting objects: 100% (18/18), done.
Delta compression using up to 8 threads
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 13.00 KiB | 1.18 MiB/s, done.
Total 18 (delta 0), reused 17 (delta 0)
To github.com:cdecker/lightning.git
* [new tag] v0.99.1 -> v0.99.1
Please ignore my comments above, I forgot to push the tag 🤦
There was a problem hiding this comment.
Yeah, no problem :)
In addition, when I ran the following command
➜ lightning git:(repobuild/v0.99.1) ✗ sudo docker run --rm -v $(pwd):/repo -ti cl-repro-bionic
I receive the following error at the end
cp: target '/repo/release/' is not a directory
and I also noted that the docs contain a path different from what you posted in the PR
In particular, you posted
release/clightning-v0.99.1-Ubuntu-18.04.tar.xz
The doc contains
/repo/release/clightning-v0.9.0rc1-Ubuntu-18.04.tar.xz
|
Was not able to reproduce. Checked out this branch. Ran These are the shas it produces. The Do I need to do any configurations to get it to work? I tried reconfiguring with rust on and rebuilding but got the same result. |
|
That is very interesting, retrying a couple of times I do get different results, and sometimes they even match your hashes: I'm wondering if it is because I had a dirty tree during the first runs. But we are cloning a clean repo as first step, then go through the tarball which matches, and build from there. I'll see if doing it from a clean tree works any more reliable. |
|
Full sequence of steps: |
|
Strange, would you mind sharing one of the ubuntu 20.04 or ubuntu 18.04 bundles? Maybe we can pin down where the diffs are. |
|
Reran the reprobuilds on a new VM generating the builder images as per instructions, and I'm getting the following results: Matching my previous results, so I'm pretty convinced now that this is the right result. |
These are mine, look different. Do you need any particular data to debug it? |
|
Ok, this is getting very strange. I took over @vincenzopalazzo's VM, trying to replicate exactly where the difference was, and I thought I had it (the builder docker image not being built with the v0.99.1 version of the Dockerfile), but it spits out different results. Then I tried to go through each step individually on both my machine and the VM, and they produce the same binaries, but different from any of the prior runs. I'm starting to believe this is something like a date thing, that is reproducible as long as the builds are close enough together, but not if done too much time apart. For reference here are the steps I used on a completely clean VM (or after deleting all the docker images that are used in the build): cd /tmp
git clone https://github.com/cdecker/lightning.git repo
cd /tmp/repo
git describe
# v0.11.0.1-239-gf67ab2a86
git checkout v0.99.1
# ...
# HEAD is now at f9bcb2510 v0.99.1
mkdir release # Directory where artifacts will land
# Build the focal base image from installation medium
sudo docker run --rm -v $(pwd):/build ubuntu:22.04 \
bash -c "apt-get update && apt-get install -y debootstrap && debootstrap focal /build/focal"
sudo tar -C focal -c . | sudo docker import - focal
# Build the builder image
sudo docker build -t cl-repro-focal - < contrib/reprobuild/Dockerfile.focal
sudo docker run --rm -v $(pwd):/repo -ti cl-repro-focalThis resulted in the following hashes on both machines: I will look into where the differences are as soon as I wrap my head around the diffoscope output. |
|
Ok, new attempt. I think I found the mismatch: dependencies updating under our feet. So now we commit the 🚧 Please notice that the |
Changelog-Added: build: Reproducible builds now include rust binaries such as the `cln-grpc` plugin
Changelog-None
We also had to switch around the directories, so now the instructions assume you're in the repo root directory
| @@ -146,6 +148,8 @@ repository (remember to checkout the tag you are trying to build): | |||
|
|
|||
| ```bash | |||
| sudo docker run --rm -v $(pwd):/repo -ti cl-repro-bionic | |||
There was a problem hiding this comment.
maybe this is a change that need to be done inside the docker container, but during the the process that start in this line I need to ran mkdir release and rerun the process again.
Maybe can be done inside the docker container or at least documented it?
|
Ok, this is my result! Let's rerun this tomorrow, and let's see what happens! |
|
ACK 6725d02 |
Keeping my fingers crossed it just works out of the box :-)
There is a dummy version with CHANGELOG and tag
v0.99.1at cdecker@f9bcb25 which can be used to check everybody agrees on the hashes: