Much higher compile times with -Z threads=8 than with -Z threads=1

[Shnatsel]

When compiling cargo audit from git on commit b6baecc0ea4e2d115e4e10b10c2196b33d42c1da, I'm seeing the project build in 19 seconds on my machine with -Z threads=1 but it takes 25 seconds with -Z threads=8.

I am using a 6-core desktop CPU, so no chance of NUMA issues. I'm also seeing 25s compile times with -Z threads=6, matching the CPU core count.

I've captured Samply profiles but they are hard to make sense of due to the sheer number of threads (4000 for a single thread, 6000 for multiple threads). They are too big for sharing via firefox.dev, so please find them attached:
profile-1-thread.json.gz
profile-8-threads.json.gz

Meta

rustc --version --verbose:

rustc 1.75.0-nightly (fdaaaf9f9 2023-11-08)
binary: rustc
commit-hash: fdaaaf9f923281ab98b865259aa40fbf93d72c7a
commit-date: 2023-11-08
host: x86_64-unknown-linux-gnu
release: 1.75.0-nightly
LLVM version: 17.0.4

[Shnatsel]

Outputs of cargo build --timings:
cargo-timing-1-thread.html.gz cargo-timing-1-thread-no-mold.html.gz
cargo-timing-8-threads.html.gz

[mjguzik]

I confirm the issue, compiling the same thing:

cargo build -r 606.59s user 33.28s system 1596% cpu 40.078 total
RUSTFLAGS="-Z threads=8" cargo build -r 771.61s user 49.88s system 1660% cpu 49.476 total

Test system has 24 cores.

rustc 1.75.0-nightly (0f44eb3 2023-11-09)
binary: rustc
commit-hash: 0f44eb3
commit-date: 2023-11-09
host: x86_64-unknown-linux-gnu
release: 1.75.0-nightly
LLVM version: 17.0.4

[Kobzol]

Do you have the same flags set for both builds? using RUSTFLAGS overrides config.toml, did you perhaps have some options there?

[mjguzik]

In my case this is a fresh clone, 0 local changes.

[Shnatsel]

I did have mold configured as the linker in config.toml. After removing the config the parallel frontend is still slower, although by not quite as much.

Single thread: 23.29s
8 threads: 25.70s

That's still a 10% regression.

[Shnatsel]

I've re-measured cargo build --timings and updated the post above to provide a correct baseline, without mold.

I still see syn v1.0.109 compilation time going from 2.3s to 7.4s, tokio v1.29.1 going up from 2.5s to 5.2s, aho-corasick v1.0.2 from 1.1s to 5.1s, and the compilation time of many other crates also increasing. See the full output of --timings for details.

[mjguzik]

I reran with RUSTFLAGS="-Z threads=1", got:

RUSTFLAGS="-Z threads=1" cargo build -r 591.51s user 30.31s system 1561% cpu 39.825 total

which is about the same as without the -Z flag.

[Shnatsel]

It's curious that a system with a higher core count is seeing a greater regression: 39.825s to 49.476 is a nearly 20% increase in compilation time, compared to a 10% increase on my 6-core CPU.

[mjguzik]

It's not particularly curious, adding more threads to a case which suffers a scalability problem does tend to increase total run time. And I have more cores to exercise the problem at the same time.

I tried to get a differential flamegraph based on perf record output, but perf report ended up executing for almost 2h(!) before I killed it, boggled down in comunicating with addr2line which kept failing to resolve anything (it was making forward progress, just incredibly slowly and the result was useless anyway). Debian 12 for interested parties.

[bjorn3]

Try perf report --no-inline. That will skip addr2line at the cost of not showing inlines functions.