Docker based reproducible builds

[orthomind]

I'm posting this pull request so that it can be reviewed. I've gone as far as I think I should without getting some feedback about how this repo is intended to be used and the approach I've taken.

These changes achieve two primary goals:

1. Method for easily building a release with one docker run command
2. Packages that produce the same sha256sum no matter whose machine they are built on (when using the docker method).

This does not work on versions that don't use dep to vendor packages. In short, it will only work from v1.1.2 onwards, so far as I can tell.

To build packages with this change you don't need to predownload any repositories. On a fresh system with Docker 17.05 or later installed, no packages fetched, you can test with:

git clone https://github.com/orthomind/decred-binaries.git
cd decred-binaries
docker build -t decred/decred-binaries .
docker run --rm -ti -e TAG="v1.2.0" -e PROD=1 -v $(pwd)/build:/build decred/decred-binaries

After that, you will find the built packages with manifest under ./build.

If you don't specify TAG, it will download the latest commit and build that. If you mount your local Go path to the docker container, it can build with your own local source. More notes in the README.

There's more changes that can be made after this as well, including enhancing tools for building without docker, ensuring that earlier versions can be built even if we change the build process later, adding tests to build and confirm manifests, and looking at other build scripts (dcrinstallbuild.sh).

[dajohi]

Please add a minimal .travis.yml file:

sudo: required
services:
  - docker

script:
  - docker build -t decred-binaries .

[orthomind]

I've added this test and expanded it to try building v1.2.0 and compare the manifest to the expected. diff returns exit status 1 if there are any differences, so it should fail if there are. However, I haven't tested to verify that travis reports a failure in such a case.

If we add more versions later as part of our build tests, then I might make a folder to store test manifests in. There's a few things that I'll need to add to support reproducible builds for multiple versions. One step at a time!

[dajohi]

Great! Last request is to format the shell scripts and README to 80 character max per line.

[orthomind]

I've limited the columns on all the files I've touched for now, and can continue to do others as I work on new files.

However, I didn't limit all the lines in README.md. Some are output from commands, and I'm not sure if you want those reduced too, as it may change the meaning. If you think they should be changed as well, let me know and I'll push up a quick commit to do so.

[dajohi]

$ docker run --rm -ti -e TAG="v1.2.0" -e PROD=1 -v $(pwd)/build:/build decred/decred-binaries
Tag set to v1.2.0
No repo found, so fetching dcrd
Cloning into 'dcrd'...
remote: Counting objects: 623, done.
remote: Compressing objects: 100% (586/586), done.
remote: Total 623 (delta 85), reused 160 (delta 27), pack-reused 0
Receiving objects: 100% (623/623), 5.65 MiB | 6.62 MiB/s, done.
Resolving deltas: 100% (85/85), done.
Note: checking out 'fd99f572cfd52ca722ad501ccb34acdcb22b4a67'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

Verifying dcrd repo tag matches v1.2.0...
Repo tag of 'release-v1.2.0' does not match specified tag of 'v1.2.0'

[orthomind]

I snuck in a fix for this with cb15b00, but I really should have done it as a separate commit. The original script I wrote didn't work when multiple tags point to the same commit.

If you pull down the latest and rebuild your docker file, it should work.

[orthomind]

@dajohi I'm planning on revisiting this shortly to see if it still works as designed, and what might need to change. Since it's been nearly a year since I submitted this PR, can you please let me know if any circumstances have changed that might impact on this?

[orthomind]

A note for myself, or if someone else decides to take over this task sometime, Go 1.13 has a new trimpath build command that should make it easier for those who want to build the same binaries without using Docker, on their own s ystem:

"The new go build flag -trimpath removes all file system paths from the compiled executable, to improve build reproducibility."

https://golang.org/doc/go1.13#go-command

[jrick]

My suggestion for reproducible builds is to drop this idea and use https://github.com/jrick/release. This will also depend on decred/decred-release#152 so that dcrinstall can be cross compiled in a similar reproducible way.

edit: spoke too soon. these builds are not necessarily reproducible. need to figure out why, because I expected with cross compilation and -trimpath they would be.

edit2: jrick/release#1

edit3: fixed!