-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compile rustc+LLVM to wasm #6
Compile rustc+LLVM to wasm #6
Conversation
Thanks a lot for this PR! I'm currently at a conference, so I probably won't get to reviewing for a couple of days. If I don't review it within a week or so feel free to ping me. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the late reply. I've rebased the wasm changes and created the new compile_rustc_for_wasm16 branch for this. Would you mind rebasing and changing the target of this PR to that branch? You can use the same edit button that allows changing the PR title and then clicking on the button next to "oligamiq wants to merge 86 commits into". Also squashing your commits into either a single commit or a logical division of commits would be appreciated.
walker.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where is this file used?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It searches for symbols to identify .c files and other files that use symbols that cannot be used in WASM. It was useful for debugging, but can be eliminated.
.define("CMAKE_CXX_COMPILER_TARGET", &wasi_target_llvm) | ||
.define("CMAKE_C_FLAGS", format!("{wasi_sysroot} {wasi_cflags_llvm}")) | ||
.define("CMAKE_CXX_FLAGS", format!("{wasi_sysroot} {wasi_cflags_llvm}")) | ||
.define("CMAKE_EXE_LINKER_FLAGS", wasi_ldflags_llvm) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't there a cmake toolchain file in the wasi-sdk that sets all these flags as necessary?
wrapper_linker_clang.sh
Outdated
shift 2 | ||
continue | ||
fi | ||
if [[ $1 == "-Wl,--max-memory=1073741824" ]]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this just remove the -Wl,
prefix? I was genuinely running out of memory without this argument. Wasm shared memories need to declare a maximum size and lld gives a very small size by default.
73bc239
to
881d67f
Compare
Since I was assuming version 15, there was a conflict. |
You can drop the first three commits that are currently in this PR when rebasing on top of compile_rustc_for_wasm16. |
I've shut down the work server, so I'll do it tomorrow. I'll give it a try. |
881d67f
to
3fe3834
Compare
Looks like the first commit is still there. If you want I could do the rebase for you and if you enable the github option for this have me push it directly to your PR branch. |
What is the 'first commit'? |
I'm referring to 295aca5. This commit originates from the compile_rustc_for_wasm15 branch of mine despite this PR now targetting the compile_rustc_for_wasm16 branch. |
If we lose it, won't it cause the build to possibly fail or fail to compile on the cranelift backend? |
Sorry. It makes me crazy because it already exists. I'll turn it off. Add Message |
d67fd0c
to
1e6418a
Compare
I believe the work is complete. |
compiler/rustc_target/src/spec/targets/wasm32_wasip1_threads.rs
Outdated
Show resolved
Hide resolved
It seems like it still isn't rebased on top of the compile_rustc_for_wasm16 branch. I've got it rebased locally and am currently building it to try it out. If you enable the option "Allow edits and access to secrets by maintainers" as explained by https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork I can push my rebased version to this PR. |
ok, understand. You want to create a new branch from wasm16 and cherry-pick it. I've given you the permissions for now, so please take care of it this time. I thought github would do it automatically... |
578bd85
to
d77d78b
Compare
Pushed
I think github preserves your choice, so if you ever disabled it, it will be kept disabled for all future PR's by default. |
No, I thought I could just apply the changes to the commit in a nice way regardless of the top of branch. |
There were some conflicts between the top of the branch and your commits, so it wouldn't have been possible to merge until those conflicts were resolved. I resolved them for you while rebasing. |
I'm getting the following locally when building LLVM:
Edit: Messed up the llvm-project submodule while I did the rebase. |
I believe there was a .define("HAVE_DLOPEN", ""); followed by a .define to turn it OFF. I'll go look for it. I have restored it, though, I understand, as I also had trouble with submodule changes. |
about [Fix linking stdc++ on wasi] config.llvm.toml: src/bootstrap/src/core/build_steps/compile.rs: |
In this case the |
3ab3533
to
ed82c66
Compare
ed82c66
to
ee85060
Compare
This is really slow when producing a 156MB rustc.wasm output.
I finally managed to get it to work locally. It does deadlock most of the time when running in wasmtime however unless I use |
really? It should have worked without any issues, but I will also test it with the copied version to ensure it works. Add: |
Updating Wasmtime seems to have fixed the deadlocks. |
oh... |
Checked that this PR doesn't break compiling cg_clif for single threaded wasm32-wasip1. I've spent enough time on this for today. I'm going to take another look at the changes tomorrow to see if anything is unnecessary or can be improved. After that I will merge this PR. Thanks again for working on this! |
This should be updated to use the https://github.com/YoWASP/llvm-project/releases/tag/rustc-2024-09-24-do-not-remove tag for the llvm-project submodule to ensure that it remains possible to build in the future. |
I think LLVM_ENABLE_THREADS=OFF was broken |
Indeed. That commit was just a couple of changes that are necessary anyway to compile for wasm32-wasip1 once LLVM_ENABLE_THREADS=OFF gets fixed. |
Also, RustCompiler have code to check if LLVM has multithreaded. |
…raheemdev Optimize `Box::default` and `Arc::default` to construct more types in place Both the `Arc` and `Box` `Default` impls currently call `T::default()` before allocating, and then moving the resulting `T` into the allocation. Most `Default` impls are trivial, which should in theory allow LLVM to construct `T: Default` directly in the `Box` allocation when calling `<Box<T>>::default()`. However, the allocation may fail, which necessitates calling `T`'s destructor if it has one. If the destructor is non-trivial, then LLVM has a hard time proving that it's sound to elide, which makes it construct `T` on the stack first, and then copy it into the allocation. Change both of these impls to allocate first, and then call `T::default` into the uninitialized allocation, so that LLVM doesn't have to prove that it's sound to elide the destructor/initial stack copy. For example, given the following Rust code: ```rust #[derive(Default, Clone)] struct Foo { x: Vec<u8>, z: String, y: Vec<u8>, } #[no_mangle] pub fn src() -> Box<Foo> { Box::default() } ``` <details open> <summary>Before this PR:</summary> ```llvm `@__rust_no_alloc_shim_is_unstable` = external global i8 ; drop_in_place() generated in case the allocation fails ; core::ptr::drop_in_place<playground::Foo> ; Function Attrs: nounwind nonlazybind uwtable define internal fastcc void `@"_ZN4core3ptr36drop_in_place$LT$playground..Foo$GT$17hff376aece491233bE"(ptr` noalias nocapture noundef readonly align 8 dereferenceable(72) %_1) unnamed_addr #0 personality ptr `@rust_eh_personality` { start: %_1.val = load i64, ptr %_1, align 8 %0 = icmp eq i64 %_1.val, 0 br i1 %0, label %bb6, label %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i" "_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i": ; preds = %start %1 = getelementptr inbounds i8, ptr %_1, i64 8 %_1.val6 = load ptr, ptr %1, align 8, !nonnull !3, !noundef !3 tail call void `@__rust_dealloc(ptr` noundef nonnull %_1.val6, i64 noundef %_1.val, i64 noundef 1) #8 br label %bb6 bb6: ; preds = %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i", %start %2 = getelementptr inbounds i8, ptr %_1, i64 24 %.val9 = load i64, ptr %2, align 8 %3 = icmp eq i64 %.val9, 0 br i1 %3, label %bb5, label %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i.i11" "_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i.i11": ; preds = %bb6 %4 = getelementptr inbounds i8, ptr %_1, i64 32 %.val10 = load ptr, ptr %4, align 8, !nonnull !3, !noundef !3 tail call void `@__rust_dealloc(ptr` noundef nonnull %.val10, i64 noundef %.val9, i64 noundef 1) #8 br label %bb5 bb5: ; preds = %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i.i11", %bb6 %5 = getelementptr inbounds i8, ptr %_1, i64 48 %.val4 = load i64, ptr %5, align 8 %6 = icmp eq i64 %.val4, 0 br i1 %6, label %"_ZN4core3ptr46drop_in_place$LT$alloc..vec..Vec$LT$u8$GT$$GT$17hb5ca95423e113cf7E.exit16", label %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i15" "_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i15": ; preds = %bb5 %7 = getelementptr inbounds i8, ptr %_1, i64 56 %.val5 = load ptr, ptr %7, align 8, !nonnull !3, !noundef !3 tail call void `@__rust_dealloc(ptr` noundef nonnull %.val5, i64 noundef %.val4, i64 noundef 1) #8 br label %"_ZN4core3ptr46drop_in_place$LT$alloc..vec..Vec$LT$u8$GT$$GT$17hb5ca95423e113cf7E.exit16" "_ZN4core3ptr46drop_in_place$LT$alloc..vec..Vec$LT$u8$GT$$GT$17hb5ca95423e113cf7E.exit16": ; preds = %bb5, %"_ZN63_$LT$alloc..alloc..Global$u20$as$u20$core..alloc..Allocator$GT$10deallocate17heaa87468709346b1E.exit.i.i.i4.i15" ret void } ; Function Attrs: nonlazybind uwtable define noalias noundef nonnull align 8 ptr `@src()` unnamed_addr #1 personality ptr `@rust_eh_personality` { start: ; alloca to place `Foo` in. %_1 = alloca [72 x i8], align 8 call void `@llvm.lifetime.start.p0(i64` 72, ptr nonnull %_1) store i64 0, ptr %_1, align 8 %_2.sroa.4.0._1.sroa_idx = getelementptr inbounds i8, ptr %_1, i64 8 store ptr inttoptr (i64 1 to ptr), ptr %_2.sroa.4.0._1.sroa_idx, align 8 %_2.sroa.5.0._1.sroa_idx = getelementptr inbounds i8, ptr %_1, i64 16 %_3.sroa.4.0..sroa_idx = getelementptr inbounds i8, ptr %_1, i64 32 call void `@llvm.memset.p0.i64(ptr` noundef nonnull align 8 dereferenceable(16) %_2.sroa.5.0._1.sroa_idx, i8 0, i64 16, i1 false) store ptr inttoptr (i64 1 to ptr), ptr %_3.sroa.4.0..sroa_idx, align 8 %_3.sroa.5.0..sroa_idx = getelementptr inbounds i8, ptr %_1, i64 40 %_4.sroa.4.0..sroa_idx = getelementptr inbounds i8, ptr %_1, i64 56 call void `@llvm.memset.p0.i64(ptr` noundef nonnull align 8 dereferenceable(16) %_3.sroa.5.0..sroa_idx, i8 0, i64 16, i1 false) store ptr inttoptr (i64 1 to ptr), ptr %_4.sroa.4.0..sroa_idx, align 8 %_4.sroa.5.0..sroa_idx = getelementptr inbounds i8, ptr %_1, i64 64 store i64 0, ptr %_4.sroa.5.0..sroa_idx, align 8 %0 = load volatile i8, ptr `@__rust_no_alloc_shim_is_unstable,` align 1, !noalias !4 %_0.i.i.i = tail call noalias noundef align 8 dereferenceable_or_null(72) ptr `@__rust_alloc(i64` noundef 72, i64 noundef 8) #8, !noalias !4 %1 = icmp eq ptr %_0.i.i.i, null br i1 %1, label %bb2.i, label %"_ZN5alloc5boxed12Box$LT$T$GT$3new17h0864de14f863a27aE.exit" bb2.i: ; preds = %start ; invoke alloc::alloc::handle_alloc_error invoke void `@_ZN5alloc5alloc18handle_alloc_error17h98142d0d8d74161bE(i64` noundef 8, i64 noundef 72) rust-lang#9 to label %.noexc unwind label %cleanup.i .noexc: ; preds = %bb2.i unreachable cleanup.i: ; preds = %bb2.i %2 = landingpad { ptr, i32 } cleanup ; call core::ptr::drop_in_place<playground::Foo> call fastcc void `@"_ZN4core3ptr36drop_in_place$LT$playground..Foo$GT$17hff376aece491233bE"(ptr` noalias noundef nonnull align 8 dereferenceable(72) %_1) rust-lang#10 resume { ptr, i32 } %2 "_ZN5alloc5boxed12Box$LT$T$GT$3new17h0864de14f863a27aE.exit": ; preds = %start ; Copy from stack to heap if allocation is successful call void `@llvm.memcpy.p0.p0.i64(ptr` noundef nonnull align 8 dereferenceable(72) %_0.i.i.i, ptr noundef nonnull align 8 dereferenceable(72) %_1, i64 72, i1 false) call void `@llvm.lifetime.end.p0(i64` 72, ptr nonnull %_1) ret ptr %_0.i.i.i } ``` </details> <details> <summary>After this PR</summary> ```llvm ; Notice how there's no `drop_in_place()` generated as well define noalias noundef nonnull align 8 ptr `@src()` unnamed_addr #0 personality ptr `@rust_eh_personality` { start: ; no stack allocation %0 = load volatile i8, ptr `@__rust_no_alloc_shim_is_unstable,` align 1 %_0.i.i.i.i.i = tail call noalias noundef align 8 dereferenceable_or_null(72) ptr `@__rust_alloc(i64` noundef 72, i64 noundef 8) #5 %1 = icmp eq ptr %_0.i.i.i.i.i, null br i1 %1, label %bb3.i, label %"_ZN5alloc5boxed16Box$LT$T$C$A$GT$13new_uninit_in17h80d6355ef4b73ea3E.exit" bb3.i: ; preds = %start ; call alloc::alloc::handle_alloc_error tail call void `@_ZN5alloc5alloc18handle_alloc_error17h98142d0d8d74161bE(i64` noundef 8, i64 noundef 72) #6 unreachable "_ZN5alloc5boxed16Box$LT$T$C$A$GT$13new_uninit_in17h80d6355ef4b73ea3E.exit": ; preds = %start ; construct `Foo` directly into the allocation if successful store i64 0, ptr %_0.i.i.i.i.i, align 8 %_8.sroa.4.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 8 store ptr inttoptr (i64 1 to ptr), ptr %_8.sroa.4.0._1.sroa_idx, align 8 %_8.sroa.5.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 16 %_8.sroa.7.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 32 tail call void `@llvm.memset.p0.i64(ptr` noundef nonnull align 8 dereferenceable(16) %_8.sroa.5.0._1.sroa_idx, i8 0, i64 16, i1 false) store ptr inttoptr (i64 1 to ptr), ptr %_8.sroa.7.0._1.sroa_idx, align 8 %_8.sroa.8.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 40 %_8.sroa.10.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 56 tail call void `@llvm.memset.p0.i64(ptr` noundef nonnull align 8 dereferenceable(16) %_8.sroa.8.0._1.sroa_idx, i8 0, i64 16, i1 false) store ptr inttoptr (i64 1 to ptr), ptr %_8.sroa.10.0._1.sroa_idx, align 8 %_8.sroa.11.0._1.sroa_idx = getelementptr inbounds i8, ptr %_0.i.i.i.i.i, i64 64 store i64 0, ptr %_8.sroa.11.0._1.sroa_idx, align 8 ret ptr %_0.i.i.i.i.i } ``` </details>
LLVM backend is now available.
It works by changing config.llvm.toml to config.toml.
This allows rustc to compile against wasm32-wasip1-threads.
https://github.com/YoWASP/clang was used as a reference.
Since wasm-ld can also be used on wasi by that project, All that remains is for browser_wasi_shim to support threading, and it should be possible to compile on the browser.
cargo is going to create a project that wraps this project like the project above, which together with rustc will generate one wasm as a driver like YoWASP/clang project.
Problem is that llvm is not patched by rust and the version of llvm is higher than rust. compiler/rustc_target/src/target_features.rs has been changed because of that.
Also, the cc library has been raised to a version that supports building to wasi.
In addition, I upgraded the wasi-sdk version.
wasm and sysroot are available at https://github.com/oligamiq/rust_wasm/
so please use them if you want to try them.
Please forgive the messy commit.