Add a Cargo-based build system to eventually replace make #31123
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
r? @brson (rust_highfive has picked a reviewer for you, use r? to override) |
|
🎊 |
|
Hero mode omg 💯 💯 💯 |
|
In terms of review, I've tried to split things up in as bite-sized chunks as possible, and I'll avoid rebasing for awhile so the commit comments are preserved |
|
Yesss ✌️ |
|
Hold the phone. |
|
@alexcrichton does it again! |
Gankra
added
T-tools
relnotes
|
Is the python bit optional? As in, if we already had a recent enough rust/cargo nightly, could we just directly build the Rust bootstrap and go from there? |
| import tarfile | ||
|
|
||
| def get(url, path, quiet=False): | ||
| if quiet: |
There was a problem hiding this comment.
Oh this is actually intended (albeit kinda weirdly) where quiet means "just print what you're downloading" and not quiet means "print the command we're running"
There's some discussion here about this as well, but I wouldn't necessarily want it to be a prerequisite to build Rust that you have Rust installed just yet. Maybe down the road we can expect that (I think gcc requires this, right?). That being said I would love to remove even the python script (as @frewsxcv has discovered I'm awful at writing python) and just have pure Rust from the get-go. For now though we require Python because of the LLVM build anyway, and I tried to keep the Python as tiny as possible. Hopefully we can either minimize the python over time (e.g. get it even smaller than it is today) or possibly even rewrite it in Rust as you mentioned and just assume that rustc is available. |
|
I've started keeping a tentative list of items left-to-implement at the top of this PR as well. I plan on explicitly not implementing them here but rather opening follow-up tracking issues for them. |
| if code != 0: | ||
| if not verbose: | ||
| print("failed to run: " + ' '.join(args)) | ||
| raise Exception("failed to run command") |
|
Aside: How does one use this build system? I tried
|
|
I'm gonna rebase this to get rid of the merge and |
This commit is the start of a series of commits which start to replace the makefiles with a Cargo-based build system. The aim is not to remove the makefiles entirely just yet but rather just replace the portions that invoke the compiler to do the bootstrap. This commit specifically adds enough support to perform the bootstrap (and all the cross compilation within) along with generating documentation. More commits will follow up in this series to actually wire up the makefiles to call this build system, so stay tuned!
During the transition period where we're still using ./configure and makefiles, read some extra configuration from `config.mk` if it's present. This means that the bootstrap build should be configured the same as the original ./configure invocation. Eventually this will all be removed in favor of only storing information in `config.toml` (e.g. the configure script will generate config.toml), but for now this should suffice.
This likely isn't always valid, and subverts auto-detection.
alexcrichton
force-pushed
the
who-doesnt-want-two-build-systems
branch
from
96bf1b9
to
1bd4a65
Compare
|
@bors: r=brson 1bd4a65db80b7ce2ab0f3c8801cfaa68009a4133 I guess that wasn't so bad after all |
These describe the structure of all our crate dependencies.
This commits adds build scripts to the necessary Rust crates for all the native dependencies. This is currently a duplication of the support found in mk/rt.mk and is my best effort at representing the logic twice, but there may be some unfortunate-and-inevitable divergence. As a summary: * alloc_jemalloc - build script to compile jemallocal * flate - build script to compile miniz.c * rustc_llvm - build script to run llvm-config and learn about how to link it. Note that this crucially (and will not ever) compile LLVM as that would take far too long. * rustdoc - build script to compile hoedown * std - script to determine lots of libraries/linkages as well as compile libbacktrace
Have all Cargo-built crates pass `--cfg cargobuild` and then add appropriate `#[cfg]` definitions to all crates to avoid linking anything if this is passed. This should help allow libstd to compile with both the makefiles and with Cargo.
This will allow it to be used as a crate in a Cargo-based build
Refactor a bit to have less repetition and #[cfg] and try to bury it all inside of a macro.
This commit adds a `--enable-rustbuild` option to the configure script which will copy a different `Makefile.in` into place to intercept all `make` invocations. Currently this makefile only has one target, but it's expected to be filled out quite a bit over time!
When building with Cargo we need to detect `feature = "jemalloc"` to enable jemalloc, so propagate this same change to the build system to pass the right `--cfg` argument.
alexcrichton
force-pushed
the
who-doesnt-want-two-build-systems
branch
from
1bd4a65
to
55dd595
Compare
|
⌛ Testing commit 55dd595 with merge 2300556... |
|
💔 Test failed - auto-mac-64-opt |
|
@bors: retry On Thu, Feb 11, 2016 at 2:16 PM, bors [email protected] wrote:
|
bors
added a commit
that referenced
this pull request
Feb 12, 2016
…, r=brson This series of commits adds the initial implementation of a new build system for the compiler and standard library based on Cargo. The high-level architecture now looks like: 1. The `./configure` script is run with `--enable-rustbuild` and other standard configuration options. 2. A `Makefile` is generate which proxies commands to the new build system. 3. The new build system has a Python script entry point which manages downloading both a Rust and Cargo nightly. This initial script also manages building the build system itself (which is written in Rust). 4. The build system, written in rust and called `bootstrap`, architects how to call `cargo` and manages building all native libraries and such. One might reasonably ask "why rewrite the build system?", which is a good question! The Rust project has used Makefiles for as long as I can remember at least, and while ugly and difficult to use are undeniably robust as they contain years worth of tweaking and tuning for working on as many platforms in as many situation as possible. The rationale behind this PR, however is: * The makefiles are impenetrable to all but a few people on this planet. This means that contributions to the build system are almost nonexistent, and furthermore if a build system change is needed it's incredibly difficult to figure out how to do so. This hindrance prevents us from doing some "perhaps fancier" things we may wish to do in make. * Our build system, while portable, is unfortunately not infinitely portable everywhere. For example the recently-introduced MSVC target is quite unlikely to have `make` installed by default (e.g. it requires building inside of an MSYS2 shell currently). Conversely, the portability of make comes at a cost of crazy and weird hacks to work around all sorts of versions of software everywhere, especially when it comes to the configure script and makefiles. By rewriting this logic in one of the most robust platforms there is, Rust, we get to assuage all of these worries for free! * There's a standard tool to build Rust crates, Cargo, but the standard library and compiler don't use it. This means that they cannot benefit easily from the crates.io ecosystem, nor can the ecosystem benefit from a standard way to build this repository itself. Moving to Cargo should help assuage both of these needs. This has the added benefit of making the compiler more approachable for newbies as working on the compiler will just happen to be working on a large Cargo project, all the same standard tools and tricks will apply. * There's a huge amount of portability information in the main distribution, for example around cross compiling, compiling on new OSes, etc. Pushing this logic into standard crates (like `gcc`) enables the community to immediately benefit from new build logic. Despite these benefits, it's going to be a long road to actually replace our current build system. This PR is just the beginning and doesn't implement the full suite of functionality as the current one, but there are many more to follow! The current implementation strategy hopes to look like: 1. Land a second build system in-tree that can be itereated on an and contributed to. This will not be used just yet in terms of gating new commits to the repo. 2. Over time, bring the second build system to feature parity with the old build system, start setting up CI for both build systems. 3. At some point in the future, switch the default to the new build system, but keep the old one around. 4. At some further point in the future, delete the entire old build system. --- Alright, so with all that out of the way, here's some more info on this PR itself. The inital build system here is contained in the `src/bootstrap` directory and just adds the necessary minimum bits to bootstrap the compiler itself. There is currently no support for building documentation, running tests, or installing, but the implemented support is: * Compiling LLVM with `cmake` instead of `./configure` + `make`. The LLVM project is removing their autotools build system, so we'd have to make this transition eventually anyway. * Compiling compiler-rt with `cmake` as well (for the same rationale as above). * Adding `Cargo.toml` to map out the dependency graph to all crates, and also adding `build.rs` files where appropriate. For example `alloc_jemalloc` has a script to build jemalloc, `flate` has a script to build `miniz.c`, `std` will build `libbacktrace`, etc. * Orchestrating all the calls to `cargo` to build the standard distribution, following the normal bootstrapping process. This also tracks dependencies between steps to ensure cross-compilation targets happen as well. * Configuration is intended to eventually be done through a `config.toml` file, so support is implemented for this. The most likely vector of configuration for now, however, is likely through `config.mk` (what `./configure` emits), so the build system currently parses this information. There's still quite a few steps left to do, and I'll open up some follow-up issues (as well as a tracking issue) for this migration, but hopefully this is a great start to get going! This PR is currently tested on all the Windows/Linux/OSX triples for x86\_64 and x86, but more portability is always welcome! --- Future functionality left to implement * [ ] Re-verify that multi-host builds work * [ ] Verify android build works * [ ] Verify iOS build work (mostly compiler-rt) * [ ] Verify sha256 and ideally gpg of downloaded nightly compiler and nightly rustc * [ ] Implement testing -- this is a huge bullet point with lots of sub-bullets * [ ] Build and generate documentation (plus the various tools we have in-tree) * [ ] Move various src/etc scripts into Rust -- not sure how this interacts with `make` build system * [ ] Implement `make install` - like testing this is also quite massive * [x] Deduplicate version information with makefiles
This was referenced Feb 12, 2016
|
🍻 |
48 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This series of commits adds the initial implementation of a new build system for
the compiler and standard library based on Cargo. The high-level architecture
now looks like:
./configurescript is run with--enable-rustbuildand other standardconfiguration options.
Makefileis generate which proxies commands to the new build system.downloading both a Rust and Cargo nightly. This initial script also manages
building the build system itself (which is written in Rust).
bootstrap, architects how tocall
cargoand manages building all native libraries and such.One might reasonably ask "why rewrite the build system?", which is a good
question! The Rust project has used Makefiles for as long as I can remember at
least, and while ugly and difficult to use are undeniably robust as they contain
years worth of tweaking and tuning for working on as many platforms in as many
situation as possible. The rationale behind this PR, however is:
planet. This means that contributions to the build system are almost
nonexistent, and furthermore if a build system change is needed it's
incredibly difficult to figure out how to do so. This hindrance prevents us
from doing some "perhaps fancier" things we may wish to do in make.
everywhere. For example the recently-introduced MSVC target is quite unlikely
to have
makeinstalled by default (e.g. it requires building inside of anMSYS2 shell currently). Conversely, the portability of make comes at a cost of
crazy and weird hacks to work around all sorts of versions of software
everywhere, especially when it comes to the configure script and makefiles.
By rewriting this logic in one of the most robust platforms there is, Rust,
we get to assuage all of these worries for free!
and compiler don't use it. This means that they cannot benefit easily from the
crates.io ecosystem, nor can the ecosystem benefit from a standard way to
build this repository itself. Moving to Cargo should help assuage both of
these needs. This has the added benefit of making the compiler more
approachable for newbies as working on the compiler will just happen to be
working on a large Cargo project, all the same standard tools and tricks will
apply.
example around cross compiling, compiling on new OSes, etc. Pushing this logic
into standard crates (like
gcc) enables the community to immediately benefitfrom new build logic.
Despite these benefits, it's going to be a long road to actually replace our
current build system. This PR is just the beginning and doesn't implement the
full suite of functionality as the current one, but there are many more to
follow! The current implementation strategy hopes to look like:
contributed to. This will not be used just yet in terms of gating new commits
to the repo.
system, start setting up CI for both build systems.
keep the old one around.
Alright, so with all that out of the way, here's some more info on this PR
itself. The inital build system here is contained in the
src/bootstrapdirectory and just adds the necessary minimum bits to bootstrap the compiler
itself. There is currently no support for building documentation, running tests,
or installing, but the implemented support is:
cmakeinstead of./configure+make. The LLVMproject is removing their autotools build system, so we'd have to make this
transition eventually anyway.
cmakeas well (for the same rationale as above).Cargo.tomlto map out the dependency graph to all crates, and alsoadding
build.rsfiles where appropriate. For examplealloc_jemallochas ascript to build jemalloc,
flatehas a script to buildminiz.c,stdwillbuild
libbacktrace, etc.cargoto build the standard distribution,following the normal bootstrapping process. This also tracks dependencies
between steps to ensure cross-compilation targets happen as well.
config.tomlfile,so support is implemented for this. The most likely vector of configuration
for now, however, is likely through
config.mk(what./configureemits), sothe build system currently parses this information.
There's still quite a few steps left to do, and I'll open up some follow-up
issues (as well as a tracking issue) for this migration, but hopefully this is a
great start to get going! This PR is currently tested on all the
Windows/Linux/OSX triples for x86_64 and x86, but more portability is always
welcome!
Future functionality left to implement
makebuild systemmake install- like testing this is also quite massive