Make -Cmetadata consistent across platforms
#14107
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit updates how `-Cmetadata` is calculated for each unit to optionally exclude the `host: ...` line that rustc prints for cross-compiled builds. Previously the full `verbose_version` was hashed for stable builds and the `host` was explicitly hashed for non-stable builds. For a build using `--target`, however, that means that the `-Cmetadata` will be different when producing the same target on different hosts (e.g. producing the binary once on Linux and once on macOS). This can hinder reproduction of a binary across different platforms even when the `--target` flag is used. For example in rust-lang/rust#117597 it was seen that a WebAssembly binary produced on different platforms was slightly different and this appears due to the differing `-Cmetadata` flags. After this commit the `-Cmetadata` flag is the same for two different platforms meaning that different platforms produce the same binary. I've tested locally and a simple project produces a different binary before this change but produces the same binary on two platforms after this change. Unfortunately automated testing of this change will be difficult since it requires two different host compilers, though.
rustbot
added
A-layout
|
Thanks! I'm not 100% certain that different host compilers will always produce the exact same output, but I imagine that should be considered a bug in rustc if it does not. @bors r+ |
bors
added
S-waiting-on-bors
|
☀️ Test successful - checks-actions |
|
A bit off topic, but how does one know when this merge will show up in the stable or nightly release? |
|
I believe cargo gets updated on nightly once a week or so. And from there it will ride the release train to stable, which can take anywhere from 6-12 weeks depending on where in the release cycle a change gets merged. |
I definitely agree yeah. Getting deterministic output across different host platforms for the same target to me is basically a big game of whack-a-mole. This PR is probably one piece of the puzzle but I also have the same hunch that there are many other pieces for completely full fidelity. |
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Jun 22, 2024
Update cargo 17 commits in 3ed207e416fb2f678a40cc79c02dcf4f936a21ce..bc89bffa5987d4af8f71011c7557119b39e44a65 2024-06-18 19:18:22 +0000 to 2024-06-22 00:36:36 +0000 - test: migrate weak_dep_features, workspaces and yank to snapbox (rust-lang/cargo#14111) - test: migrate features and features(2|_namespaced) to snapbox (rust-lang/cargo#14100) - test: Add auto-redaction for not found error (rust-lang/cargo#14124) - test: migrate build to snapbox (rust-lang/cargo#14068) - test: migrate unit_graph, update and vendor to snapbox (rust-lang/cargo#14119) - fix(test): Un-redact Packaged files (rust-lang/cargo#14123) - test: Auto-redact file number (rust-lang/cargo#14121) - test: migrate lints_table and lints/(mod|unknown_lints) to snapbox (rust-lang/cargo#14104) - Simplify checking feature syntax (rust-lang/cargo#14106) - test: migrate testsuites to snapbox (rust-lang/cargo#14091) - Make `-Cmetadata` consistent across platforms (rust-lang/cargo#14107) - fix(toml): Warn when edition is unuset, even when MSRV is unset (rust-lang/cargo#14110) - Add `CodeFix::apply_solution` and impl `Clone` (rust-lang/cargo#14092) - test: migrate `cargo_alias_config&cargo_config/mod` to snapbox (rust-lang/cargo#14093) - Simplify checking for dependency cycles (rust-lang/cargo#14089) - test: Migrate `pub_priv.rs` to snapshot (rust-lang/cargo#14103) - test: migrate rustdoc and rustdocflags to snapbox (rust-lang/cargo#14098) <!-- r? ghost -->
github-actions bot
pushed a commit
to rust-lang/miri
that referenced
this pull request
Jun 23, 2024
Update cargo 17 commits in 3ed207e416fb2f678a40cc79c02dcf4f936a21ce..bc89bffa5987d4af8f71011c7557119b39e44a65 2024-06-18 19:18:22 +0000 to 2024-06-22 00:36:36 +0000 - test: migrate weak_dep_features, workspaces and yank to snapbox (rust-lang/cargo#14111) - test: migrate features and features(2|_namespaced) to snapbox (rust-lang/cargo#14100) - test: Add auto-redaction for not found error (rust-lang/cargo#14124) - test: migrate build to snapbox (rust-lang/cargo#14068) - test: migrate unit_graph, update and vendor to snapbox (rust-lang/cargo#14119) - fix(test): Un-redact Packaged files (rust-lang/cargo#14123) - test: Auto-redact file number (rust-lang/cargo#14121) - test: migrate lints_table and lints/(mod|unknown_lints) to snapbox (rust-lang/cargo#14104) - Simplify checking feature syntax (rust-lang/cargo#14106) - test: migrate testsuites to snapbox (rust-lang/cargo#14091) - Make `-Cmetadata` consistent across platforms (rust-lang/cargo#14107) - fix(toml): Warn when edition is unuset, even when MSRV is unset (rust-lang/cargo#14110) - Add `CodeFix::apply_solution` and impl `Clone` (rust-lang/cargo#14092) - test: migrate `cargo_alias_config&cargo_config/mod` to snapbox (rust-lang/cargo#14093) - Simplify checking for dependency cycles (rust-lang/cargo#14089) - test: Migrate `pub_priv.rs` to snapshot (rust-lang/cargo#14103) - test: migrate rustdoc and rustdocflags to snapbox (rust-lang/cargo#14098) <!-- r? ghost -->
This was referenced Jul 25, 2024
|
Adding the The Rust compiler now uses the same metadata for symbol ABI compatibility regardless of the host the Rust compiler was compiled for. This makes it possible for cross-compiles to the same target from different hosts to produce identical binaries. (There may still be other issues with doing so, which we hope to resolve over time.) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit updates how
-Cmetadatais calculated for each unit to optionally exclude thehost: ...line that rustc prints for cross-compiled builds. Previously the fullverbose_versionwas hashed for stable builds and thehostwas explicitly hashed for non-stable builds. For a build using--target, however, that means that the-Cmetadatawill be different when producing the same target on different hosts (e.g. producing the binary once on Linux and once on macOS).This can hinder reproduction of a binary across different platforms even when the
--targetflag is used. For example in rust-lang/rust#117597 it was seen that a WebAssembly binary produced on different platforms was slightly different and this appears due to the differing-Cmetadataflags. After this commit the-Cmetadataflag is the same for two different platforms meaning that different platforms produce the same binary.I've tested locally and a simple project produces a different binary before this change but produces the same binary on two platforms after this change. Unfortunately automated testing of this change will be difficult since it requires two different host compilers, though.