Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tests failing with multiple matching crates for serde #4015

Closed
phansch opened this issue Apr 22, 2019 · 2 comments · Fixed by #4017
Closed

Tests failing with multiple matching crates for serde #4015

phansch opened this issue Apr 22, 2019 · 2 comments · Fixed by #4017
Labels
C-bug Category: Clippy is not doing the correct thing

Comments

@phansch
Copy link
Member

phansch commented Apr 22, 2019

rust-lang/rust#60053 broke the Clippy tests. Specifically tests/ui/serde.rs fails with:

error[E0464]: multiple matching crates for `serde`
  --> $DIR/serde.rs:4:1
   |
LL | extern crate serde;
   | ^^^^^^^^^^^^^^^^^^^
   |
   = note: candidates:
           crate `serde`: /home/<user>/code/rust-clippy/target/debug/deps/libserde-3ec33c1c405dbd9a.rlib
           crate `serde`: /home/<user>/.rustup/toolchains/master/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde-4b8bb77bdaea80db.rlib

error[E0463]: can't find crate for `serde`
  --> $DIR/serde.rs:4:1
   |
LL | extern crate serde;
   | ^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0463`.

I'm not really sure how fix this. Maybe somewhere in compiletest-rs? The error still happens with a clean Rust master installation. Can we somehow ignore the sysroot serde?

cc Manishearth/compiletest-rs#114, rust-lang/rust#24853
@phansch phansch added the C-bug Category: Clippy is not doing the correct thing label Apr 22, 2019
@Xanewok
Copy link
Member

Xanewok commented Apr 22, 2019 •
edited
Loading

First of all, sorry for the breakage.

I'll also paste the Rust CI logs (albeit for a diferent case):

[01:25:18] +error[E0464]: multiple matching crates for `serde`
[01:25:18] +  --> $DIR/used_underscore_binding_macro.rs:11:10
[01:25:18] +   |
[01:25:18] +LL | #[derive(Deserialize)]
[01:25:18] +   |          ^^^^^^^^^^^
[01:25:18] +   |
[01:25:18] +   = note: candidates:
[01:25:18] +           crate `serde`: /checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde-bb2ee07afa357acf.rlib
[01:25:18] +           crate `serde`: /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps/libserde-2ef066c730b0470a.rlib
[01:25:18] +
[01:25:18] +error[E0463]: can't find crate for `_serde`
[01:25:18] +  --> $DIR/used_underscore_binding_macro.rs:11:10
[01:25:18] +   |
[01:25:18] +LL | #[derive(Deserialize)]
[01:25:18] +   |          ^^^^^^^^^^^ can't find crate
[01:25:18] +
[01:25:18] +error: aborting due to 2 previous errors
[01:25:18] +
[01:25:18] +For more information about this error, try `rustc --explain E0463`.
[01:25:18] +
[01:25:18] 
[01:25:18] The actual stderr differed from the expected stderr.
[01:25:18] Actual stderr saved to /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base/crashes/used_underscore_binding_macro.stderr
[01:25:18] To update references, run this command from build directory:
[01:25:18] tests/ui/update-references.sh '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base' 'crashes/used_underscore_binding_macro.rs'
[01:25:18] 
[01:25:18] error: 1 errors occurred comparing output.
[01:25:18] status: exit code: 1
[01:25:18] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools-bin/clippy-driver" "tests/ui/crashes/used_underscore_binding_macro.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base/crashes/used_underscore_binding_macro.stage-id" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps" "-Dwarnings" "-Zui-testing" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/x86_64-unknown-linux-gnu/release/build/clippy-77007bd62c0dadf4/out/test_build_base/crashes/used_underscore_binding_macro.stage-id.aux" "-A" "unused"

As I understand it, this is because we have 2 version of serde:

  • one in the sysroot (compiler uses its own version) at
/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde-bb2ee07afa357acf.rlib
  • one for the tools at
/checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps/libserde-2ef066c730b0470a.rlib

which is pulled for the compiletest compiler invocation (wrapped in a clippy_driver) with a
-L /checkout/obj/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps

I don't have a clear idea how to approach this, so to think out loud:

Obviously we can't drop the -L because clippy-driver needs their deps to run,
...but also we're interested in the serde used in the local build,
...hence we'd like to ignore sysroot serde somehow.

I don't think we can ever let local deps coalesce to the ones used in the compiler, because they may be semver-incompatible and compiler would like to have its own thread-locals and such.

Could we somehow "blacklist" sysroot crates on a lib-searching level if a rustc_private feature isn't enabled? However, if that's enabled (which I believe is for Clippy) should we set a precedence? There are both cases where sysroot is preferred and also where -L ones are preferred...
Could we differentiate sysroot crates on a semantic level (e.g. use sysroot::serde)? (rust-lang/rust#57288)

I hope solving this would also enable using serde_derive from crates.io in the compiler (same problem with shadowed sysroot libs via -L), which we had to workaround by manually expanding the macros in the rls-data (not pretty!)

FWIW RLS and Clippy work correctly with Rust patch on crates that pull serde dependency (tested in the RLS). Worst case, I can revert the patch but would really like to see a path forward with this.

EDIT: It seems we can disambiguate these by using explicit --extern flag

@Xanewok
Copy link
Member

Xanewok commented Apr 22, 2019

Indeed, with the original invocation of

LD_LIBRARY_PATH=$(rustc --print sysroot)/lib "target/debug/clippy-driver" "tests/ui/serde.rs" "-L" "/home/xanewok/repos/rust-clippy/target/debug/test_build_base" "--target=x86_64-unknown-linux-gnu" "-C" "prefer-dynamic" "-o" "/home/xanewok/repos/rust-clippy/target/debug/test_build_base/serde.stage-id" "-L" "target/debug" "-L" "target/debug/deps" "-Dwarnings" "-Zui-testing" "-L" "/home/xanewok/repos/rust-clippy/target/debug/test_build_base/serde.stage-id.aux" "-A" "unused"

I get the following error:

error[E0464]: multiple matching crates for `serde`
  --> tests/ui/serde.rs:4:1
   |
LL | extern crate serde;
   | ^^^^^^^^^^^^^^^^^^^
   |
   = note: candidates:
           crate `serde`: /home/xanewok/repos/rust-clippy/target/debug/deps/libserde-66800e6ad9385298.rlib
           crate `serde`: /home/xanewok/repos/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde-bb2ee07afa357acf.rlib

error[E0463]: can't find crate for `serde`
  --> tests/ui/serde.rs:4:1
   |
LL | extern crate serde;
   | ^^^^^^^^^^^^^^^^^^^ can't find crate

but if I add the disambiguating --extern

--extern serde=/home/xanewok/repos/rust-clippy/target/debug/deps/libserde-66800e6ad9385298.rlib

this correctly compiles.

@Xanewok Xanewok mentioned this issue Apr 22, 2019
Merged
bors added a commit that referenced this issue Apr 23, 2019
@bors
Auto merge of #4017 - Xanewok:wip-disambiguate-extern, r=oli-obk
6ae46a8 
compiletest: Disambiguate extern crate deps shared with the compiler

Fixes #4015.

changelog: Handle deps shared with the compiler in the internal compiletest suite

Attempts to fix the multiple matching crates error using the `--extern dep=path` disambiguation. This only includes `serde` at the moment because it's the only problematic dep right now (is inside Rust sysroot and pulled via `extern crate` in the test suite).

I'm not exactly sure this is the right approach (FWIW it fixes the issue locally), please do tell if this should be done differently.
@dtolnay dtolnay mentioned this issue Jun 30, 2019
Closed
@Xanewok Xanewok mentioned this issue Sep 7, 2019
Closed
@tesuji tesuji mentioned this issue Feb 1, 2020
Merged
bors added a commit that referenced this issue Feb 5, 2020
@bors
Auto merge of #5121 - lzutao:comptest, r=flip1995
185b11f 
Improve compile_test and dogfood tests

* Hopefully this finally resolves the "multiple matching crates" error, e.g. #4015
* This handle some convenient CARGO env like CARGO_TARGET_DIR and CARGO_BUILD_TARGET

changelog: none
bors added a commit that referenced this issue Feb 5, 2020
@bors
Auto merge of #5121 - lzutao:comptest, r=flip1995
45eba89 
Improve compile_test and dogfood tests

* Hopefully this finally resolves the "multiple matching crates" error, e.g. #4015
* This handle some convenient CARGO env like CARGO_TARGET_DIR and CARGO_BUILD_TARGET

changelog: none
bors added a commit that referenced this issue Feb 6, 2020
@bors
Auto merge of #5121 - lzutao:comptest, r=flip1995
c1f8be0 
Improve compile_test and dogfood tests

* Hopefully this finally resolves the "multiple matching crates" error, e.g. #4015
* This handle some convenient CARGO env like CARGO_TARGET_DIR and CARGO_BUILD_TARGET

changelog: none
@flip1995 flip1995 mentioned this issue Feb 8, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Category: Clippy is not doing the correct thing
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

compiletest: Disambiguate extern crate deps shared with the compiler Xanewok/rust-clippy
2 participants
@phansch @Xanewok