Implement the Cargo half of pipelined compilation (take 2)

[alexcrichton]

This commit starts to lay the groundwork for #6660 where Cargo will
invoke rustc in a "pipelined" fashion. The goal here is to execute one
command to produce both an *.rmeta file as well as an *.rlib file
for candidate compilations. In that case if another rlib depends on that
compilation, then it can start as soon as the *.rmeta is ready and not
have to wait for the *.rlib compilation.

Initially attempted in #6864 with a pretty invasive refactoring this
iteration is much more lightweight and fits much more cleanly into
Cargo's backend. The approach taken here is to update the
DependencyQueue structure to carry a piece of data on each dependency
edge. This edge information represents the artifact that one node
requires from another, and then we a node has no outgoing edges it's
ready to build.

A dependency on a metadata file is modeled as just that, a dependency on
just the metadata and not the full build itself. Most of cargo's backend
doesn't really need to know about this edge information so it's
basically just calculated as we insert nodes into the DependencyQueue.
Once that's all in place it's just a few pieces here and there to
identify compilations that can be pipelined and then they're wired up
to depend on the rmeta file instead of the rlib file.

Closes #6660

[rust-highfive]

r? @nrc

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

r? @ehuss

So I've got one open question on this implementation which is included as a failing test here. The general question is what do we want to happen in this scenario:

• In the first compile, a user builds an rlib.
• In the next compile, the user builds another rlib that depends on the first.

The test executes this as cargo build -p dependency followed by cargo build -p parent. In this scenario it's currently buggy because the second compilation assumes the first produced metadata, but we currently try to avoid producing metadata where possible. The options to solve this are:

1. Always produce metadata files for rlibs, whether or not the metadata will be consumed during pipelining
2. In the second compile recompile the dependency rlib to produce metadata. (we'd handle this by hashing whether metadata is produced into the fingerprint)

Option (1) would take up a bit more disk space while option (2) may take longer, but this seems unlikely to really come up in practice all that often. I'd slightly lean towards option (1), but I'm curious what you'd think as well?

[bors]

☔ The latest upstream changes (presumably #6772) made this pull request unmergeable. Please resolve the merge conflicts.

[Eh2406]

Are these the same metadata files need for 'cargo check' to use the results of 'cargo build'?

[alexcrichton]

I think in theory they can be compatible, but this doesn't attempt to set things up so cargo check can be shared with cargo build, but we can perhaps tackle that not too difficultly in the future!

[ehuss]

I'd slightly lean towards option (1), but I'm curious what you'd think as well?

Either option sounds fine to me. It doesn't seem like a huge amount of extra disk space.

[alexcrichton]

Ok, I've implemented that strategy since it's easier!

[ehuss]

Looks like a bunch of tests need to be updated now that --emit has been expanded.

[alexcrichton]

Oops, updated that

[ehuss]

Looks good to me. r=me if you're ready to go. I'm not sure if you want to wait for the rustc changes or not. I figure worst case if things don't work out, this can be reverted?

[alexcrichton]

I think it makes sense to wait for an end-to-end story to prove out that it's working. To that end I've pushed up a few more commits which I believe fully implements pipelined compilation in Cargo given the support in upstream rustc for -Z emit-directives. Some caveats apply though:

• This'll fail CI since we don't have a nightly which supports emit-directives
• This'll have very little benefit for pipelined compilation since rustc doesn't emit the directive until the very end of compilation. The compiler is being changed in Emit metadata files earlier rust#60385 to emit the directive much sooner
• Still requires opt in, for example through CARGO_BUILD_PIPELINING=1

[alexcrichton]

Ok I think that covers everything and I've verified it works with tonights nightly, so this should be good to go!

[bors]

☔ The latest upstream changes (presumably #6896) made this pull request unmergeable. Please resolve the merge conflicts.

[alexcrichton]

@ehuss did you want to give this a final once-over before merging for the commits I added after the initial review?

[ehuss]

I'm still working through this. I don't see any noticeable improvement in total compilation time with the various projects I've tried. Is there a particular project type or concurrency level that would see a benefit? I'm concerned about moving forward if it ultimately doesn't help.

I've been instrumenting this a little. I've been building some graphs of the queue sizes, which shows that the initial phase is kept more "full", but that drops off quickly. I've also been recording how much faster the rmeta files are emitted. On average, the rmeta file is emitted at about the 70% mark of the total crate build time (ranging from 30% to 98%). I'm not sure if that intuitively seems reasonable. I believe it is currently emitted just before codegen?

I'm going to continue instrumenting and try to get a better picture. Would like to get your perspective.

[alexcrichton]

Heh a good point on the investigations! My plan was moreso "land it and make a post asking for users to test it out" and debug from there :)

I was also slightly dismayed at the lack of benefit on Cargo itself in both release and debug builds. I suspect that this is not going to be as large of a win as I had previously hoped. That being said, I'm still hopeful we can get testers on internals to help show some other data.

This also isn't necessarily a win in all cases. The whole point of pipelining is to take advantage of otherwise idle parallelism, and we can't pipeline literally everything and what's here is a coarse approximation of the ideal. Some ingredients necessary to reap benefits from this PR:

• You'll at least have to compile more than one crate. Incremental builds that build the top crate will see no benefit.
• You'll probably need to be compiling in release mode to get the most benefit. Otherwise codegen is typically quick enough that it won't really feel all that different. (AKA if codegen is "fast enough" in debug mode pipelining won't actually start anything that much sooner).
• There may not be a ton of benefit in full crate graph builds. There's lots of edges that still require full synchronization of build process, such as:
  • Procedural macros (aka serde_derive) have to be entirely finished before proceeding
  • Build scripts (aka building large chunks of C code) must entirely finish as well

Given all that I suspect that the scenario with the biggest win will be incremental release mode builds where a crate a few layers deep in the graph was modified. Incremental builds in debug mode may fit this same profile but will likely not feel all that much faster.

FWIW though I'm personally pretty happy with how low the complexity level turned out for this PR, so I think it'd be fine to land with the intention of backing it out if it doesn't pan out. That being said though I can't think of a use case where this regresses something, so I'm not sure there's a ton of downside to not landing this. (that's exclusively given the low complexity budget spend I'm perceiving here)

[stuhood]

I was also slightly dismayed at the lack of benefit on Cargo itself in both release and debug builds. I suspect that this is not going to be as large of a win as I had previously hoped. That being said, I'm still hopeful we can get testers on internals to help show some other data.

From simulations we've done with pipelining/"outlining", the benefits only begin to show on beefier machines with larger graphs. If you don't have enough total work, or if you don't have enough cores to run more of the unblocked work in parallel, then the outlining is overhead (but hopefully a very small amount of overhead).

A case where outlining helps even for smaller workspaces is the "long path" case. In a workspace with a chain of dependencies, if you edit a few deps away from a test/binary, the "path" between them can be compiled in parallel, whereas without outlining it would be sequential.

...So yea. I suspect that @alexcrichton's hypotheses are correct.

[ehuss]

Sounds reasonable.

I'd like to add some timing data to the debug logs, I'll follow up with a separate PR with some ideas.

I think this will address some other issues, like #6674 and #6119.

but it turns out that we unconditionally capture all output from the compiler/rustdoc today, no exceptions

This hasn't been true since #6289. EDIT: Or maybe this means that the stdout/stderr are not inherited anymore, nevermind.

[alexcrichton]

Updated with @ehuss's comments.

@matthiaskrgr I'm not sure why, but I'm not really shooting for 100% fidelity works everywhere in this PR, that's something I hope we can fix and iterate on afterwards.

[ehuss]

Sorry, one more minor nit. I notice the progress bar jumping backwards at times. I think the following will fix it:

diff --git a/src/cargo/core/compiler/job_queue.rs b/src/cargo/core/compiler/job_queue.rs
index bff2d5b6..10ecf47f 100644
--- a/src/cargo/core/compiler/job_queue.rs
+++ b/src/cargo/core/compiler/job_queue.rs
@@ -320,7 +320,8 @@ impl<'a, 'cfg> JobQueue<'a, 'cfg> {
             // unnecessarily.
             let events: Vec<_> = self.rx.try_iter().collect();
             let events = if events.is_empty() {
-                self.show_progress(total);
+                let count = total - self.queue.len() - self.active.len() - queue.len();
+                self.show_progress(count, total);
                 vec![self.rx.recv().unwrap()]
             } else {
                 events
@@ -431,8 +432,7 @@ impl<'a, 'cfg> JobQueue<'a, 'cfg> {
         }
     }

-    fn show_progress(&mut self, total: usize) {
-        let count = total - self.queue.len() - self.active.len();
+    fn show_progress(&mut self, count: usize, total: usize) {
         let active_names = self
             .active
             .values()

Essentially when jobs enter the local queue, they get lost from the count until they get added to active. It also made the progress bar appear to be further along than it actually was.

EDIT: Alternatively, it could keep a running "finished" count.

[alexcrichton]

Ah yeah I think I like the idea of a running finished count, so pushed that!

[ehuss]

I think this needs to be moved down to the Artifact::All branch?

[alexcrichton]

Hm yes indeed

[ehuss]

👍

@bors r+

[bors]

📌 Commit dcd7c48 has been approved by ehuss

[bors]

⌛ Testing commit dcd7c48 with merge 2e09266...

[bors]

☀️ Test successful - checks-travis, status-appveyor
Approved by: ehuss
Pushing 2e09266 to master...

[mark-i-m]

So is pipelining implemented on nightly now?

[Eh2406]

No. It is on master for cargo. Every week or so there is a PR on rust to update the cargo subrepo. The next nightly after that PR is merged will have this functionality. Then the functionality still requires an opt in with the env CARGO_BUILD_PIPELINING.

[alexcrichton]

I've posted an internals topic for getting more information here