Dockerized LLVM

[jackcmay]

Problem

BPF programs will soon require customized LLVM toolchain

Summary of Changes

Sync and build LLVM in docker and with 2 stage docker image make the binaries available to run directly. This docker is optionally used to build BPF programs.

Further changes are coming to populate this docker with the llvm binaries build and released separately.

Fixes #

[mvines]

looks good so far, I see the WIP label so I'll refrain from nits or looking too hard

[jackcmay]

@mvines. The WIP is more to sync up with what you have in mind. This is working but is setup so that we run llvm from within Docker. With the Travis building and posting binaries in release tags would we still be building in a docker?

[borromeotlhs]

@jackcmay Running llvm in docker is loosely defined, as you are right: travis would be creating the binaries. However, do you have any ways to verify a built toolchain? As if anyone is building it, the docker file would just download it from there. . .

This type of thing is admirable, but I think after this is committed, the resultant binary (docker container) should be tagged and pushed to dockerhub to be repulled for veracity's sake against the version you successfully built.

Deeper conversation should be had about dedicating the toolchains to a repository and the levels of veracity one places on correctly building to a known good standard. I have not see gpg checksums on built toolchains unless one is downloading them directly or building them from source like bitbake or LEDE on Openwrt.

@mvines is there any discussion for ground up veracity on the resultant blockchain? I wonder what portions you all are building besides the applications? I assume the blockchain computation backend needs compiling?

[mvines]

With the Travis building and posting binaries in release tags would we still be building in a docker?

Probably not, ideally we can use the native binaries directly. But the docker setup still has value in the short term, in that we'd likely first get Travis building Linux LLVM binaries, so macOS could continue to use the docker image until we get Travis building macOS LLVM binaries. For Windows support I feel that we can just create a new issue for it and one day somebody may be motivated enough to pick it up.

[jackcmay]

@borromeotlhs
We are building the programs (smart contracts) with LLVM. We need custom changes to LLVM to support things like BPF shared objects. Up until this point, we've been downloading publically available LLVM binaries. In this PR we are building our own forks of LLVM. To make this easier we are doing this in docker and will be pushing the resulting docker images to docker hub.

[mvines]

probably should document somewhere around here that the user needs to docker pull solanalabs/llvm

[jackcmay]

Doesn't docker do that automagically if it does not find the docker locally?

[mvines]

nerp

[jackcmay]

How about pull as part of the script or is that just wasteful?

#!/usr/bin/env bash
set -ex
SDKPATH="$( cd "$(dirname "$0")" ; pwd -P )"/../../../..
docker pull solanalabs/llvm
docker run --workdir /solana_sdk --volume $SDKPATH:/solana_sdk --rm solanalabs/llvm `basename "$0"` "$@"

[mvines]

ci/docker-run.sh does this, but there's also cases when you don't want to pull (you're on an airplane with no uplink) so if we autopull then we'd also want to add a way to disable it. My vote for now is to just document that the user must pull

[mvines]

r+ with doc nits picked