Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass SOURCE_DATE_EPOCH to docker run #128

Merged
merged 2 commits into from
Sep 24, 2024

Conversation

segiddins
Copy link
Contributor

See rubygems/rubygems#2289, it helps make gem builds reproducible

See rubygems/rubygems#2289, it helps make gem builds reproducible
@segiddins
Copy link
Contributor Author

Ah I think build/sudoers also might need to be updated?

@flavorjones
Copy link
Collaborator

I've added a test and included it in the sudoers allowlist.

@flavorjones
Copy link
Collaborator

@segiddins Assuming this goes green, can you suggest some documentation improvements to help gem maintainers use this feature?

@segiddins
Copy link
Contributor Author

Hmm. It's not something that's necessary for folks to use while building, but it is a pre-requisite to be able to rebuild a gem (because of timestamps in the tarball, which RubyGems will take from SOURCE_DATE_EPOCH, and also timestamps in the linked binary, and ld also respects SOURCE_DATE_EPOCH)

@flavorjones
Copy link
Collaborator

@segiddins Sorry, I'm not sure I understand your last response. Who is this feature for, what will they do with it, how will they use it? I think this is necessary both for user documentation but also so that I understand why this code is there as the maintainer.

@ianks
Copy link

ianks commented Sep 7, 2024

@flavorjones This shouldn’t be a feature that users really need to know about, but it’s a key tenant of reproducible builds.. Certain OS, like nix, mandate its usage.

@flavorjones
Copy link
Collaborator

It still needs to be documented and I'd like at least for someone to verify that this works as expected.

@flavorjones
Copy link
Collaborator

My point here really is that we don't know if the feature actually works to generate reproducible gems. I wrote a test that the env var gets exported. And I know what the env var does. But does the whole stack work?

If nobody is willing to try it out and tell me it works, and explain how to build gems with it, it doesn't seem worth merging right now.

@segiddins
Copy link
Contributor Author

Running CI=1 bundle exec rake gem:x86_64-linux-musl in test/rcd_test before this change twice (and mv pkg pkg-1 after the first invocation), and diffing the results

❯ diffoscope --max-diff-block-lines-saved 100 --exclude-directory-metadata yes pkg-1 pkg
--- pkg-1
+++ pkg
│   --- pkg-1/rcd_test-1.0.0-x86_64-linux-musl.gem
├── +++ pkg/rcd_test-1.0.0-x86_64-linux-musl.gem
│ ├── file list
│ │ @@ -1,3 +1,3 @@
│ │ --r--r--r--   0 wheel        (0) wheel        (0)      599 2024-09-24 15:44:57.000000 metadata.gz
│ │ --r--r--r--   0 wheel        (0) wheel        (0)     7298 2024-09-24 15:44:57.000000 data.tar.gz
│ │ --r--r--r--   0 wheel        (0) wheel        (0)      295 2024-09-24 15:44:57.000000 checksums.yaml.gz
│ │ +-r--r--r--   0 wheel        (0) wheel        (0)      599 2024-09-24 15:45:21.000000 metadata.gz
│ │ +-r--r--r--   0 wheel        (0) wheel        (0)     7295 2024-09-24 15:45:21.000000 data.tar.gz
│ │ +-r--r--r--   0 wheel        (0) wheel        (0)      297 2024-09-24 15:45:21.000000 checksums.yaml.gz
│ ├── metadata.gz
│ │ ├── filetype from file(1)
│ │ │ @@ -1 +1 @@
│ │ │ -gzip compressed data, last modified: Tue Sep 24 15:44:57 2024, max compression, from Unix
│ │ │ +gzip compressed data, last modified: Tue Sep 24 15:45:21 2024, max compression, from Unix
│ ├── data.tar.gz
│ │ ├── filetype from file(1)
│ │ │ @@ -1 +1 @@
│ │ │ -gzip compressed data, last modified: Tue Sep 24 15:44:57 2024, max compression, from Unix
│ │ │ +gzip compressed data, last modified: Tue Sep 24 15:45:21 2024, max compression, from Unix
│ │ ├── data.tar
│ │ │ ├── file list
│ │ │ │ @@ -1,14 +1,14 @@
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)      569 2024-09-24 15:44:57.000000 ext/java/RcdTestExtService.java
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)      528 2024-09-24 15:44:57.000000 ext/java/RubyRcdTest.java
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)     4815 2024-09-24 15:44:57.000000 ext/mri/extconf.rb
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)     1697 2024-09-24 15:44:57.000000 ext/mri/rcd_test_ext.c
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)      216 2024-09-24 15:44:57.000000 ext/mri/rcd_test_ext.h
│ │ │ │ --rw-r--r--   0 wheel        (0) wheel        (0)      132 2024-09-24 15:44:57.000000 lib/rcd_test.rb
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/2.4/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/2.5/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/2.6/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/2.7/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/3.0/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/3.1/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/3.2/rcd_test_ext.so
│ │ │ │ --rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:44:57.000000 lib/rcd_test/3.3/rcd_test_ext.so
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)      569 2024-09-24 15:45:21.000000 ext/java/RcdTestExtService.java
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)      528 2024-09-24 15:45:21.000000 ext/java/RubyRcdTest.java
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)     4815 2024-09-24 15:45:21.000000 ext/mri/extconf.rb
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)     1697 2024-09-24 15:45:21.000000 ext/mri/rcd_test_ext.c
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)      216 2024-09-24 15:45:21.000000 ext/mri/rcd_test_ext.h
│ │ │ │ +-rw-r--r--   0 wheel        (0) wheel        (0)      132 2024-09-24 15:45:21.000000 lib/rcd_test.rb
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/2.4/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/2.5/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/2.6/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/2.7/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/3.0/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/3.1/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/3.2/rcd_test_ext.so
│ │ │ │ +-rwxr-xr-x   0 wheel        (0) wheel        (0)     5920 2024-09-24 15:45:21.000000 lib/rcd_test/3.3/rcd_test_ext.so
│ ├── checksums.yaml.gz
│ │ ├── filetype from file(1)
│ │ │ @@ -1 +1 @@
│ │ │ -gzip compressed data, last modified: Tue Sep 24 15:44:57 2024, max compression, from Unix
│ │ │ +gzip compressed data, last modified: Tue Sep 24 15:45:21 2024, max compression, from Unix
│ │ ├── checksums.yaml
│ │ │ @@ -1,7 +1,7 @@
│ │ │  ---
│ │ │  SHA256:
│ │ │ -  metadata.gz: 745f6ecb74370f44bae4383c3c71e07d99802ff35f2a630fee912608bc5d60c9
│ │ │ -  data.tar.gz: c589c08c5cf9e42506681506453ee1632c1ace37a7957fd08b8077e76c0a27b6
│ │ │ +  metadata.gz: 8c5622a20327150fcf3f48cc7cc8e06c719bc93a365dfea2da5fc957dec12e1f
│ │ │ +  data.tar.gz: ab96733d2dcc104c1e8aa32a3702c756cf46b844a9f3231b25eade931ed38160
│ │ │  SHA512:
│ │ │ -  metadata.gz: 8cac7391d46ea0b4ee8a5ac99a0e52bc6d1c0ad8d5cb842f0ef5909c247172fad9b20063711fcc48f323e293e991d7b963f95cf0a1f54e63307669796f52f046
│ │ │ -  data.tar.gz: f87986e4ce87077a3a968388877fe8e10b47943750ecf312bb8a0c1296997846390c649ec299fd8deddb16b95448efd15e57667a9dbbca8fec2610116366c68d
│ │ │ +  metadata.gz: 37bf64cf1e7bc67c75787628e53abd78d19a7b672dddaef4aaf56c8aba5bf70192360ba33a1921cd68857722c69fdf81484dfb38f5dfbfbba864167225f2cfd2
│ │ │ +  data.tar.gz: e5a820b22e7c7f2afa910d82f9f6446f2fcba2004caaab0fd117ab641d7a3934ecf59606f8ab0bfb72d18daed296e9ff66aac5b06d7ac0dbdba00c3523a5af66

after

❯ sha256sum pkg*/*
sha256sum: pkg-1/rcd_test-1.0.0-x86_64-linux-musl: Is a directory
175acd6f5f345f11bd5b6ca071e728cf3ba8eb75333dfd18f94100bc11dda02f  pkg-1/rcd_test-1.0.0-x86_64-linux-musl.gem
sha256sum: pkg/rcd_test-1.0.0-x86_64-linux-musl: Is a directory
175acd6f5f345f11bd5b6ca071e728cf3ba8eb75333dfd18f94100bc11dda02f  pkg/rcd_test-1.0.0-x86_64-linux-musl.gem

So setting the source date epoch is enough (for a simple gem) to get a byte-for-byte identical build out of rake-compiler-dock. In normal use, you'd extract the SOURCE_DATE_EPOCH from the built gem (by looking at the timestamps in the .gem tar file), and then only set it on the rebuild

@flavorjones
Copy link
Collaborator

@segiddins Great! Thank you for exercising the stack.

I realize I must now be completely annoying you, but I feel you're missing an opportunity to educate people on what reproducible builds are and how to use this tool as part of that build chain by not adding something to the README. 🤷 But I'll stop asking and merge the feature, undocumented.

If you decide in the future that you want to add something to the README, even if it's a link pointing to something you (or someone else) has written, I would gladly merge it.

@flavorjones flavorjones merged commit 8308b0e into rake-compiler:main Sep 24, 2024
172 checks passed
@segiddins
Copy link
Contributor Author

I agree it's a great idea to document reproducible gem builds 😅 cc @duckinator, we should do that for the gem rebuild command you added

@segiddins segiddins deleted the patch-1 branch September 24, 2024 17:12
@duckinator
Copy link

Yeah, definitely.

It's not proper documentation, but there's a very rough WIP tutorial about it at https://pup-e.com/resources/reproducible-gem-builds/. It doesn't discuss the inner workings at all, just the user-facing parts, but it should be enough to try it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants