cargo build sometimes causes overrides to be built when it is already built
#1215
Comments
|
Do you have an example override I could peek at? I haven't personally seen this crop up yet. |
|
my Projects in question: Root project that depended on the other ones: |
|
Can you also try running a build with |
|
Please let me step in, as I am experiencing the same right now. It's new to me, must have sneaked in recently.
Metarustc --version --verbose
rustc 1.0.0-nightly (522d09dfe 2015-02-19) (built 2015-02-19)
binary: rustc
commit-hash: 522d09dfecbeca1595f25ac58c6d0178bbd21d7d
commit-date: 2015-02-19
build-date: 2015-02-19
host: x86_64-apple-darwin
release: 1.0.0-nightlycargo --version --verbose
cargo 0.0.1-pre-nightly (43755c0 2015-02-19) (built 2015-02-19) |
|
@Byron do you have a series of steps which reproduces this behavior? Unfortunately I'm not sure that the logs are enough to help diagnose this right now :( |
|
@alexcrichton I will try to cook something up - you should have it in an hour or so, provided reproducing this works out for me. |
mkdir tmp && cd tmp
git clone https://github.com/hyperium/hyper
git clone https://github.com/Byron/yup-oauth2
# Make sure that the full path to 'hyper' is in your ~/.cargo/config paths array
cd yup-oauth2
# build tests ...
cargo test
# This might not rebuild
cargo test
# keep calling, it won't be long until it rebuilds hyper
...As I used the steps above to reproduce the issue, I am confident you will manage to do the same. |
|
Aha, I have reproduced! I am still baffled, but I'm at least a little closer! |
|
Terrific ! Good luck hunting that thing down mercilessly ! |
|
Hm ok, this is definitely quite odd. In @bryon's example it looks like @Byron I can't seem to reproduce today (the compiler has changed), can you still reproduce this? Some other helpful logs would be |
|
Me neither (even with an updated example, as the |
|
This is happening to me again, since upgrading to version It may be worth noting that the main difference is that cargo will always recompile the program's main-dependency, as well as the program itself, for example: ➜ google-apis-rs git:(master) ✗ make discovery1-cli-cargo ARGS=build
cd gen/discovery1-cli && cargo build
Compiling google-discovery1 v0.1.7+00000000 (file:///Users/byron/Documents/dev/rust/google-apis-rs/gen/discovery1-cli)
Compiling google-discovery1-cli v0.2.0+00000000 (file:///Users/byron/Documents/dev/rust/google-apis-rs/gen/discovery1-cli)
# Now running it again will perform the same build, even though nothing changed
➜ google-apis-rs git:(master) ✗ make discovery1-cli-cargo ARGS=build
cd gen/discovery1-cli && cargo build
Compiling google-discovery1 v0.1.7+00000000 (file:///Users/byron/Documents/dev/rust/google-apis-rs/gen/discovery1-cli)
Compiling google-discovery1-cli v0.2.0+00000000 (file:///Users/byron/Documents/dev/rust/google-apis-rs/gen/discovery1-cli)Thus, it is not happening for overrides, but for the entire program. In my particular case this causes enormous build overheads, as I deal with 78 programs that now want to recompile all the time. |
|
I think the key difference to other projects is that, in this case, As this might be unrelated, I created a new issue. |
|
I noticed that dependencies fetched from crates.io or github are cached, but any local dependencies aren't, regardless of whether the path to a local dependency is relative or not. Now, what's interesting is that for remore dependencies the fingerprint file seems okay (http://pastebin.com/dFEUvQVs) while for the local dependencies it contains some gibberish on every line (http://pastebin.com/p0rnm0KG)! cargo 0.3.0-nightly (06dbe65 2015-05-29) (built 2015-05-31) |
|
@ArtemGr hm that may actually be a different issue, a dependency on something like |
|
Only |
|
Do you have a project I could take a peek at? |
|
Yeah, I can upload it but I'm not sure there's a point. The yup-oauth2 mentioned above have a similar issue and you can observe a similar fingerprint gibberish: If yup-oauth2 is fixed but I still has a problem then I'll make a separate test case. That all right with you? Note that I'm using an older rustc nightly ( |
|
Hm that looks like a bug in |
|
@ArtemGr could you also gist the logs of |
|
Aha, then it definitely looks like your bug is a dupe of serde-rs/serde#85. It looks like @Byron's logs don't indicate that bug, however, so I'm still at a loss for what this is. |
|
Thanks! Hopefully fixing the Serde's bug will make it easier to triage the Byron's issue. |
|
If I were to patch a Cargo fork with a temporary workaround in order to filter the bogus dependencies, where in the Cargo should I look to do that? |
|
Much appreciated! |
|
I'm experiencing this problem as well, so I thought I would add a sample point. I have two crates A and B and B depends on A. If I start an empty project and add either A or B as the only dependency - no problem. However, when I add both, the behavior of random rebuilds starts happening. I am experiencing this specifically when A is |
|
Yup, the small test case was completely blank with these lines added to Cargo.toml: However, as per your suggestion, I changed this to: This fixes the problem. The fix holds up in my actual project too, not just the distilled test case. Thanks! |
|
@alexcrichton Actually, my original test case seems to be resolved simply with more recent versions of Cargo (a bisect pointed to 40d1c10 and #1567). Sorry about that, updating Cargo should have been the first thing I did. |
|
Ah no worries, glad it's fixed! |
|
So here's - https://github.com/ArtemGr/cargo/commit/f56369511c2737c5c18a63e382f5bbbae88f8de4 - a patch that fixes it for me. Proves that in my case serde-deprecated/quasi#7 is the culprit. I wonder if we should ignore a dependency if it does not exists? |
This means that a source file may have been deleted, in which case we need to re-run the compiler to really make sure that it is in fact not needed. |
This commit is an architectural change inside of Cargo itself in the way that it handles the output format of builds. Previously when a build start, all existing directories and files would be renamed to `old-foo` folders. The build would then `rename` all files back into the right location as they were seen as fresh and needed for the build. The benefit of a system such as this is a rock-solid guarantee that the build tree contains exactly what it would if we were to start the build from a totally clean directory each time. There are some downsides, however: * In #800, it was discovered that this method has an unfortunate interaction with Docker. Docker apparently will mount many filesystems which `rename` will not work across. * I have seen countless flaky failures on windows due to an attempt to remove a file that was still in use somehow. I've never been able to truly track down why these failures are happening, however. The new system for managing output files is to build up a list of all known files at the start of a build, whitelist any necessary files when the build is being prepared, and then wipe out all unknown files right before the build begins. This is not quite as close to the guarantee as the benefits reaped before because on the second build all build files will still be in their final output locations, they may just get updated as part of the build as well. This seems like an acceptable compromise, however. Closes #800
|
In my particular situation I am building multiple projects in parallel into the same target-dir. This works fine, until cargo thinks it has to rebuild a perfectly up-to-date dependency, and deletes it, which in turn causes the other build processes to fail. The last time this happened, it looked like this. Cargo really wants to rebuild items it has just build, including Unfortunately, the issue faded away before I could have a look at the contents of the fingerprints file. How to reproduceEven though the following statements would get you close to my setup, it differs as it will also use rustc. In my local setup, I use multirust with the stable toolchain, but manually symlinked cargo from nightly into the stable toolchain. Also when trying these statements myself, I was unable to reproduce the issue, which might possibly hint at rustc being involved in some way. $ git clone https://github.com/Byron/google-apis-rs
$ cd google-apis-rs
# need nightly to get latest cargo - rustc stable would be the default way to compile google.rs
$ multirust override nightly
# build the first library without any make-controlled parallelism to get all dependencies
$ make discovery1-cli-cargo ARGS=build
# now build APIs and CLIs in parallel. It takes about 30m to show the issue presented in the gist.
# I wonder why something is deleted all of the sudden ... .
$ time make cargo-cli ARGS=build -j8 |
|
Thanks for the heads-up - I will post some thoughts in the referenced issue. |
|
I have a feeling that this is probably a dupe of #2041, so I'm going to close in favor of that. |
cargo builda bunch of times. sometimes it unnecessarily rebuilds the overridden dependency.This can be a serious issue if the compile time for the dependency is long.
The text was updated successfully, but these errors were encountered: