Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up tidy #105829

Merged
merged 3 commits into from
Dec 18, 2022
Merged

Speed up tidy #105829

merged 3 commits into from
Dec 18, 2022

Conversation

the8472
Copy link
Member

@the8472 the8472 commented Dec 17, 2022

This can be reviewed commit by commit since they contain separate optimizations.

# master
$ taskset -c 0-5 hyperfine './x test tidy'
Benchmark #1: ./x test tidy
  Time (mean ± σ):      4.857 s ±  0.064 s    [User: 12.967 s, System: 2.014 s]
  Range (min … max):    4.779 s …  4.997 s    10 runs

# PR
$ taskset -c 0-5 hyperfine './x test tidy'
Benchmark #1: ./x test tidy
  Time (mean ± σ):      3.672 s ±  0.035 s    [User: 10.524 s, System: 2.029 s]
  Range (min … max):    3.610 s …  3.725 s    10 runs

@rustbot
Copy link
Collaborator

rustbot commented Dec 17, 2022

r? @jyn514

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 17, 2022
Copy link
Member

@jyn514 jyn514 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this :) I don't mind merging as-is, but the concurrency code here is getting pretty hairy, I wonder if it makes sense to switch to rayon.

Comment on lines +290 to +304
if line.is_empty() {
if i == 0 {
leading_new_lines = true;
}
trailing_new_lines += 1;
continue;
} else {
trailing_new_lines = 0;
}

let trimmed = line.trim();

if !trimmed.starts_with("//") {
lines += 1;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did this move? I don't see lines / trailing_new_lines / leading_new_lines used until after where it was before.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we can continue earlier and skip attempting all the string processing stuff below on empty lines.

Comment on lines +38 to +44
let drain_handles = |handles: &mut VecDeque<ScopedJoinHandle<'_, ()>>| {
// poll all threads for completion before awaiting the oldest one
for i in (0..handles.len()).rev() {
if handles[i].is_finished() {
handles.swap_remove_back(i).unwrap().join().unwrap();
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this useful? Does it run any faster or is it just releasing the resources earlier?

Oh, I guess removing a handle means you have more space in the queue to add more jobs? I'm a little confused why you would only run this once instead of putting it in the loop below ... maybe we should use rayon::ParallelIterator instead so we always join the first job to finish? All the custom concurrency code is a little hard to maintain in general.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is called every time in the check macro when a task is added to make room for the next task.

When I originally wrote this code (before is_finished existed) I think I wanted to add rayon but someone said it's better to keep the number of dependencies low so it's faster when one doesn't have tidy compiled.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, the only place I see rayon mentioned in the original PR is #81833 (comment) - do you remember why you made that change?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, looks like #81833 (comment). Ok, we don't need to revisit that here.

@jyn514
Copy link
Member

jyn514 commented Dec 17, 2022

@bors r+

@bors
Copy link
Contributor

bors commented Dec 17, 2022

📌 Commit ab7d769 has been approved by jyn514

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 17, 2022
bors added a commit to rust-lang-ci/rust that referenced this pull request Dec 17, 2022
…iaskrgr

Rollup of 8 pull requests

Successful merges:

 - rust-lang#104854 (Symlink `build/host` -> `build/$HOST_TRIPLE`)
 - rust-lang#105458 (Allow blocking `Command::output`)
 - rust-lang#105559 (bootstrap: Allow installing `llvm-tools`)
 - rust-lang#105789 (rustdoc: clean up margin CSS for scraped examples)
 - rust-lang#105792 (docs: add long error explanation for error E0320)
 - rust-lang#105814 (Support call and drop terminators in custom mir)
 - rust-lang#105829 (Speed up tidy)
 - rust-lang#105836 (std::fmt: Use args directly in example code)

Failed merges:

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit cf08eda into rust-lang:master Dec 18, 2022
@rustbot rustbot added this to the 1.68.0 milestone Dec 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants