Build rustc from source, fix a few issues with the RTS build system #2519
Conversation
|
Is it ok if I have a closer look possibly only after launch? |
Sure, no rush. |
|
I asked some questions about this in Rust user forum: https://users.rust-lang.org/t/cargo-vendor-and-zbuild-std-dont-work-together/59388 |
|
Having a brief look, but here the build fails with It looks like there is a build step that tries to download code (which isn’t kosher in a hermetic build environment)? |
|
Ah, that is rust trying to download a compiled rust compiler to bootstrap. Building a compiler isn’t trivial, I have doubts if “let’s write our own derivation” will be worth it – just look at https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/rust/rustc.nix, we’d probably rediscover the reason and solution for most of the logic there |
|
I wonder if with NixOS/nixpkgs@5a923e5 we can actually use a the |
to see what breaks, and also to get NixOS/nixpkgs@5a923e5 which may allow us to use `rustc` from `nixpkgs` again (see #2519)
Just updating the compiler is easy, but I also want to make sure we will always use LLVM 10 (which is the version we can deal with in our linker). Anything else will cause compile or runtime errors. I just checked the rustc.nix you linked. If I'm reading it right, https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/rust/rustc.nix#L2 these parameters allow linking rustc with a given LLVM, so this derivation should be good enough. |
What breaks with later versions with our linker? |
|
mo-ld can only link code generated by LLVM 10. Any other version will cause problems in compile time (maybe in runtime too, not sure). I tried with LLVM 11 (used by recent versions of rustc) and mo-ld was unable to link it. I don't remember the error message now. |
|
Well, maybe it’s easier fixing our tools to keep up than work hard to build against old versions? I noticed that there are more subsections to the |
This stuff is not even documented. It was basically trial and error + reverse engineering to make it link LLVM 10 generated code. Until Wasm has a better (standard, documented) linking story I don't think we will be able to keep up with LLVM releases. |
I don't remember if the issue was names section. I think I should have a comment in an issue/PR that shows at least some of the errors, but not sure where.. |
It’s not that bad: We are implementing https://github.com/WebAssembly/tool-conventions/blob/master/DynamicLinking.md, so there is documentation. In my experiments I was often facing problems with |
to see what breaks, and also to get NixOS/nixpkgs@5a923e5 which may allow us to use `rustc` from `nixpkgs` again (see #2519) Issues encountered: * [x] in nixpks, various `cargoSHA256` were wrong. (NixOS/nixpkgs#123522, NixOS/nixpkgs#123348, NixOS/nixpkgs#123349) * [x] Ocaml built with musl, used for static builds, is currently broken. Reported at NixOS/nixpkgs#124476. Found a possible fix, submitted at NixOS/nixpkgs#124498 and currently included in this branch. * [x] New ocamlformat version * [x] Running new formatter on these files * [x] The RTS fails to build or link the C files (`stddef.h` missing) This stuff was very hairy before, and that doesn’t get better :-( If I use the wrapped `wasm32-unknown-wasi-clang`, the build makes some progress, but then fails with https://bugs.llvm.org/show_bug.cgi?id=43393 (as it used to before, and then somehow we fixed that) In the end, surgically adding an include to `CLANG_FLAGS` works, it seems. * [x] Some upgraded tool produces new `names` subsections, presumably from this Wasm proposal: https://github.com/WebAssembly/extended-name-section/blob/master/proposals/extended-name-section/Overview.md Worked around by jumping over them, but otherwise ignoring them. Should be refined. * [x] Haskell code ought to be adjusted to latest GHC * [x] Bump the `niv` that we install into the shell (also because of Haskell changes) * [x] The darwin rebuild of `rustc` and `git` keeps timing out on hydra.
* Use nixpkgs master to see what breaks, and also to get NixOS/nixpkgs@5a923e5 which may allow us to use `rustc` from `nixpkgs` again (see #2519) * Use buildDunePackage * Less stdenv.lib * Update generated nix files * Bump to get fixes * Bump nixpkgs * Bump ocamlformat * Add nixpkgs patch * More patches * Try building vlq differently * Fix vlq * Build vlq with buildDunePackage and dune2 * Revert "Build vlq with buildDunePackage and dune2" This reverts commit d5e3399. * Disable static ocaml build * Allow requisites for now * Quench warning * More warnings * Better error message * Try GHC-8.8.4 * Actually bump to GHC-8.10.4 * Bump niv * Fix parsing global names * Call bindgen directly * Bump nixpkgs, patches have been applied * Run formatter * Does this work? * More comments * Try static biuld again * Revert "Try static biuld again" This reverts commit ab44bbf. * Actually follow the release branch * Include patch from NixOS/nixpkgs#124498 * More patches * Set allowedRequisites again * Remove references to menhir * Refresh patch * Upgrade LLVM to version 12 * Apply suggestions from code review Co-authored-by: Claudio Russo <[email protected]> * Start decoding extended name sections * Export apply_global_relocs in RTS module, call it before RTS init * Make __wasm_apply_data_relocs optional in the linker (but not in codegen!), update linker test outputs * Call wasm_apply_global_relocs from link_start * Put the comment back * Add patch from https://github.com/NixOS/nixpkgs/pull/125472/files * Patch now upstream * Linker: remove special case around __wasm_apply_global_relocs, add RTS start to merged module start * Prepend m2 start in join_modules Co-authored-by: Claudio Russo <[email protected]> Co-authored-by: Ömer Sinan Ağacan <[email protected]>
|
Closing this in favor of #2761 |
With this PR we build rustc and cargo from the source, with the LLVM version we
support in the linker and we use to build
stubs.c(LLVM 10).The main problem this fixes is: we can now update rustc without having to worry
about rustc using a different LLVM version than
and breaking linking, or worse.
In addition, newer Rusts support building core, std, and alloc crates, so we
also ditch Xargo and use cargo (fixes #2251).
Not immediately needed, but having more control over how we build the compiler
might also be useful in the future. The previous derivation would download
pre-build nightly compiler from Rust's binary distribution site. We now build
it from source (including cargo, but not rustfmt).
Current status
I can enter
nix-shell, build the RTS. All tests pass.nix-builddoesn't work yet.TODOs and questions
I can't build rustfmt in this derivation. This should be a bug in the Rust
repo, but I'm not sure. I should try different commits to see if any of them
is able to build rustfmt.
nix-buildfails withI think I need to update one of the
Cargo.lockfiles, but I'm not sure how.To make sure we only use one version of Rust toolchain we should update
nixpkgs.rustPlatform-nightlyto use the toolchain we've built. Currently weuse different toolchain for the vendoring the RTS dependencies and building
the RTS.