Overhaul FileSearch and SearchPaths
#56090
Conversation
rust-highfive
added
the
S-waiting-on-review
nnethercote
force-pushed
the
filesearch
branch
from
c06f658
to
6a04166
Compare
|
Here are some instruction count results. The wall-time numbers are roughly similar (as far as I can tell given how noisy they are). This suggests that the reading of the directory metadata isn't expensive compared to the allocations. |
|
Nice find! This code in particular is sort of insanely old, not really having changed much since 2012 basically. I think @eddyb is right in that we could probably fix this in a more first-class fashion perhaps? I'm thinking something like:
That way in theory we just have a borrowed iterator of paths floating around which should allocate far less? |
|
I initially tried to put the cache in the So then I tried
rust/src/librustc_codegen_llvm/back/link.rs Lines 214 to 221 in f1e2fa8 In that case there is no So I'm sympathetic to wanting this to be cleaner, but I don't see how to do it. The key question is: where should the cache be stored? I can't see a better place than |
|
@alexcrichton Pretty sure search paths are created at session creation time from the command-line arguments and never modified afterwards. So we should be able to run all of this on all search paths at the same time. I was thinking we'd have an "init cache at the first use" policy, but if doing it eagerly, always, seems reasonable (which I think it is, because you always end up needing them), I'm definitely for it.
|
|
Ok, I can generate the cache when |
nnethercote
force-pushed
the
filesearch
branch
from
6a04166
to
a8ca7f8
Compare
|
I have updated the code to put the cache (well, it's more of a map now) on the |
|
@bors try |
|
⌛ Trying commit a8ca7f800ff2d3730cca0faca4a3a3fb4d90deaf with merge e9c0d9989e4d8ae3428c17a629a42d5f7504db1c... |
|
This is an aside about performance, for @rust-lang/wg-compiler-performance. I re-measured performance locally on the So I looked in detail at Things to note:
How can that happen? Let's do a diff with Cachegrind. Here is the top part of the diff for a Check build: It's mostly changes you'd expect from this PR: Now, here's the top part of the diff for a Debug build: What a mess. And it's all LLVM stuff, swamping the |
|
☀️ Test successful - status-travis |
|
@rust-timer build e9c0d9989e4d8ae3428c17a629a42d5f7504db1c |
|
Success: Queued e9c0d9989e4d8ae3428c17a629a42d5f7504db1c with parent 910ec6d, comparison URL. |
|
Finished benchmarking try commit e9c0d9989e4d8ae3428c17a629a42d5f7504db1c |
nnethercote
force-pushed
the
filesearch
branch
from
a8ca7f8
to
af1aaf5
Compare
|
@eddyb: A new, map-free version is up. |
nnethercote
force-pushed
the
filesearch
branch
from
af1aaf5
to
6ad4b3c
Compare
Overhaul `FileSearch` and `SearchPaths` `FileSearch::search()` traverses one or more directories. For each directory it generates a `Vec<PathBuf>` containing one element per file in that directory. In some benchmarks this occurs enough that the allocations done for the `PathBuf`s are significant, and in practice a small number of directories are being traversed over and over again. For example, when compiling the `tokio-webpush-simple` benchmark, two directories are traversed 58 times each. Each of these directories have more than 100 files. We can do all the necessary traversals up front, when `Session` is created, and get the `Vec<PathBuf>`s then. This reduces instruction counts on several benchmarks by 1--5%. r? @alexcrichton CC @eddyb, @michaelwoerister, @nikomatsakis
|
@nnethercote I don't think |
|
💔 Test failed - status-travis |
bors
added
S-waiting-on-review
|
The job Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
Instead of maybe storing its own sysroot and maybe deferring to the one in `Session::opts`, just clone the latter when necessary so one is always directly available. This removes the need for the getter.
It's more idiomatic, makes the code shorter, and will help with the next commit.
`FileSearch::search()` traverses one or more directories. For each directory it generates a `Vec<PathBuf>` containing one element per file in that directory. In some benchmarks this occurs enough that the allocations done for the `PathBuf`s are significant, and in practice a small number of directories are being traversed over and over again. For example, when compiling the `tokio-webpush-simple` benchmark, two directories are traversed 58 times each. Each of these directories have more than 100 files. This commit changes things so that all the `Vec<PathBuf>`s that will be needed by a `Session` are precomputed when that `Session` is created; they are stored in `SearchPath`. `FileSearch` gets a reference to the necessary `SearchPath`s. This reduces instruction counts on several benchmarks by 1--5%. The commit also removes the barely-used `visited_dirs` hash in `for_each_lib_searchPath`. It only detects if `tlib_path` is the same as one of the previously seen paths, which is unlikely.
Returning an iterator leads to nicer code all around.
|
This is the error: Which is somehow hit during rustdoc processing. That error message has been seen multiple times in the past, e.g. #42867, https://bugzilla.mozilla.org/show_bug.cgi?id=1457583. Updating rustc or sccache or something else has at times made it go away again. This PR in theory does not change functional behaviour, other than it causes directories to be traversed once instead of multiple times. @alexcrichton, @glandium: any ideas? If not, my plan is to do a clumsy bisection by landing the commits one at a time in separate PRs, so I can at least narrow the problem down to a single commit. |
|
Er sorry, I meant to help out taking a look into this earlier! That error message typically means that there's a preexisting bug somewhere else that ended up being discovered here. Knowing about that error message and looking at this patch, my guess is that this bug only surfaces during doctests where we're running multiple instances of rustc in parallel in the same process (multiple Now all that being said if that panic is hit then it's a bug in the configuration of the build system. Something above rustc is telling it "hey look at these file descriptors for synchronization" but those file descriptors are actually invalid. Typically rustbuild, when building the compiler, removes relevant env vars to ensure Cargo has control over everything (and we rarely inherit proper configuration in the build system. I think the bug here though is that this is happening during doctests that aren't spawned by Cargo, but rather |
In an attempt to avoid "thread '<unnamed>' panicked at 'failed to acquire jobserver token: Bad file descriptor" errors.
nnethercote
force-pushed
the
filesearch
branch
from
4b70b01
to
209240d
Compare
|
@alexcrichton: Goodness! I will take your word for it. I've added a new commit, does that match your expectations? |
|
I swear I have reasons for my unreasonable demands! @bors r=eddyb |
|
📌 Commit 209240d has been approved by |
bors
added
S-waiting-on-bors
Overhaul `FileSearch` and `SearchPaths` `FileSearch::search()` traverses one or more directories. For each directory it generates a `Vec<PathBuf>` containing one element per file in that directory. In some benchmarks this occurs enough that the allocations done for the `PathBuf`s are significant, and in practice a small number of directories are being traversed over and over again. For example, when compiling the `tokio-webpush-simple` benchmark, two directories are traversed 58 times each. Each of these directories have more than 100 files. We can do all the necessary traversals up front, when `Session` is created, and get the `Vec<PathBuf>`s then. This reduces instruction counts on several benchmarks by 1--5%. r? @alexcrichton CC @eddyb, @michaelwoerister, @nikomatsakis
|
☀️ Test successful - status-appveyor, status-travis |
|
Hurray it worked! |
|
Holy smokes, it did! |
FileSearch::search()traverses one or more directories. For eachdirectory it generates a
Vec<PathBuf>containing one element per filein that directory.
In some benchmarks this occurs enough that the allocations done for the
PathBufs are significant, and in practice a small number ofdirectories are being traversed over and over again. For example, when
compiling the
tokio-webpush-simplebenchmark, two directories aretraversed 58 times each. Each of these directories have more than 100
files.
We can do all the necessary traversals up front, when
Sessionis created,and get the
Vec<PathBuf>s then.This reduces instruction counts on several benchmarks by 1--5%.
r? @alexcrichton
CC @eddyb, @michaelwoerister, @nikomatsakis