Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tidy compiles very slowly #123002

Closed
matthiaskrgr opened this issue Mar 24, 2024 · 7 comments · Fixed by #123339
Closed

tidy compiles very slowly #123002

matthiaskrgr opened this issue Mar 24, 2024 · 7 comments · Fixed by #123339
Assignees
@onur-ozkan
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@matthiaskrgr
Copy link
Member

matthiaskrgr commented Mar 24, 2024 •
edited by scottmcm
Loading

  1. modify some code of tidy
  2. run ./x.py test
    =>
    for the first ~6 seconds, tidy is being built in parallel, but for the remaining ~1 minute and 9 seconds llvm is doing some sequential work.

I'm not sure what takes so long, there must be one epic cgu that it is processing or something?
@matthiaskrgr matthiaskrgr added the C-bug Category: This is a bug. label Mar 24, 2024
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Mar 24, 2024
@matthiaskrgr matthiaskrgr added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) and removed C-bug Category: This is a bug. labels Mar 24, 2024
@onur-ozkan
Copy link
Member

I wanted to check the problem but couldn't reproduce it locally.

@matthiaskrgr
Copy link
Member Author 

   Compiling tidy v0.1.0 (/home/matthias/vcs/github/rust/src/tools/tidy)
time:   0.000; rss:   44MB ->   45MB (   +1MB)	parse_crate
time:   0.000; rss:   45MB ->   46MB (   +1MB)	setup_global_ctxt
time:   0.000; rss:   47MB ->   47MB (   +0MB)	crate_injection
time:   0.019; rss:   48MB ->   84MB (  +37MB)	expand_crate
time:   0.019; rss:   48MB ->   84MB (  +37MB)	macro_expand_crate
time:   0.000; rss:   84MB ->   84MB (   +0MB)	AST_validation
time:   0.000; rss:   85MB ->   85MB (   +0MB)	finalize_macro_resolutions
time:   0.004; rss:   85MB ->   88MB (   +4MB)	late_resolve_crate
time:   0.000; rss:   88MB ->   89MB (   +0MB)	resolve_check_unused
time:   0.000; rss:   89MB ->   89MB (   +0MB)	resolve_postprocess
time:   0.005; rss:   84MB ->   89MB (   +4MB)	resolve_crate
time:   0.001; rss:   96MB ->   96MB (   +0MB)	drop_ast
time:   0.011; rss:   89MB ->   96MB (   +7MB)	looking_for_derive_registrar
time:   0.013; rss:   89MB ->   97MB (   +9MB)	misc_checking_1
time:   0.023; rss:   97MB ->  129MB (  +32MB)	coherence_checking
time:   0.139; rss:   97MB ->  158MB (  +61MB)	type_check_crate
time:   0.143; rss:  158MB ->  201MB (  +43MB)	MIR_borrow_checking
time:   0.026; rss:  201MB ->  201MB (   +0MB)	MIR_effect_checking
time:   0.011; rss:  202MB ->  202MB (   +1MB)	module_lints
time:   0.011; rss:  202MB ->  202MB (   +1MB)	lint_checking
time:   0.004; rss:  202MB ->  202MB (   +0MB)	privacy_checking_modules
time:   0.016; rss:  201MB ->  202MB (   +1MB)	misc_checking_3
time:   0.091; rss:  202MB ->  214MB (  +12MB)	generate_crate_metadata
time:   0.084; rss:  214MB ->  220MB (   +5MB)	monomorphization_collector_graph_walk
time:   0.021; rss:  220MB ->  225MB (   +5MB)	partition_and_assert_distinct_symbols
time:   0.972; rss:  225MB ->  409MB ( +184MB)	codegen_to_LLVM_IR
time:   1.080; rss:  214MB ->  409MB ( +194MB)	codegen_crate
time:   0.016; rss:  409MB ->  355MB (  -54MB)	free_global_ctxt
time:  52.535; rss:  249MB ->  532MB ( +283MB)	LLVM_passes
time:   0.011; rss:  523MB ->  216MB ( -307MB)	join_worker_thread
time:  51.617; rss:  355MB ->  216MB ( -139MB)	finish_ongoing_codegen
time:   0.270; rss:  216MB ->  217MB (   +1MB)	link_rlib
time:   0.074; rss:  217MB ->  217MB (   +0MB)	link_binary_remove_temps
time:   0.344; rss:  216MB ->  217MB (   +1MB)	link_binary
time:   0.344; rss:  216MB ->  216MB (   +0MB)	link_crate
time:  51.961; rss:  355MB ->  216MB ( -139MB)	link
time:  53.517; rss:   32MB ->  186MB ( +154MB)	total
time:   0.000; rss:   44MB ->   45MB (   +1MB)	parse_crate
time:   0.000; rss:   46MB ->   46MB (   +1MB)	setup_global_ctxt
time:   0.000; rss:   47MB ->   48MB (   +0MB)	crate_injection
time:   0.009; rss:   48MB ->   76MB (  +28MB)	expand_crate
time:   0.000; rss:   76MB ->   76MB (   +0MB)	check_unused_macros
time:   0.009; rss:   48MB ->   76MB (  +28MB)	macro_expand_crate
time:   0.000; rss:   76MB ->   77MB (   +0MB)	AST_validation
time:   0.000; rss:   77MB ->   77MB (   +0MB)	finalize_imports
time:   0.002; rss:   77MB ->   79MB (   +2MB)	late_resolve_crate
time:   0.000; rss:   79MB ->   79MB (   +0MB)	resolve_check_unused
time:   0.000; rss:   79MB ->   80MB (   +0MB)	resolve_postprocess
time:   0.002; rss:   77MB ->   80MB (   +3MB)	resolve_crate
time:   0.000; rss:   80MB ->   80MB (   +0MB)	complete_gated_feature_checking
time:   0.001; rss:   80MB ->   82MB (   +2MB)	looking_for_entry_point
time:   0.002; rss:   80MB ->   83MB (   +3MB)	misc_checking_1
time:   0.001; rss:   83MB ->   87MB (   +4MB)	coherence_checking
time:   0.030; rss:   83MB ->  111MB (  +29MB)	type_check_crate
time:   0.023; rss:  111MB ->  123MB (  +12MB)	MIR_borrow_checking
time:   0.004; rss:  123MB ->  124MB (   +1MB)	MIR_effect_checking
time:   0.002; rss:  124MB ->  126MB (   +2MB)	module_lints
time:   0.003; rss:  124MB ->  126MB (   +2MB)	lint_checking
time:   0.003; rss:  124MB ->  126MB (   +2MB)	misc_checking_3
time:   0.061; rss:  126MB ->  153MB (  +27MB)	monomorphization_collector_graph_walk
time:   0.010; rss:  153MB ->  155MB (   +2MB)	partition_and_assert_distinct_symbols
time:   0.000; rss:  156MB ->  159MB (   +3MB)	write_allocator_module
time:   0.437; rss:  165MB ->  288MB ( +122MB)	codegen_to_LLVM_IR
time:   0.513; rss:  126MB ->  288MB ( +161MB)	codegen_crate
time:   0.012; rss:  288MB ->  238MB (  -50MB)	free_global_ctxt
time:   1.769; rss:  189MB ->  273MB (  +85MB)	LLVM_passes
time:   0.001; rss:  228MB ->  225MB (   -2MB)	join_worker_thread
time:   1.367; rss:  238MB ->  226MB (  -12MB)	finish_ongoing_codegen
time:   0.879; rss:  226MB ->  226MB (   +0MB)	run_linker
time:   0.884; rss:  226MB ->  226MB (   +1MB)	link_binary
time:   0.884; rss:  226MB ->  226MB (   +1MB)	link_crate
time:   2.251; rss:  238MB ->  226MB (  -12MB)	link
time:   2.859; rss:   32MB ->  152MB ( +120MB)	total
    Finished `release` profile [optimized + debuginfo] target(s) in 56.66s

@matthiaskrgr
Copy link
Member Author

perf
🤔

@matthiaskrgr
Copy link
Member Author

aha lol!
it boils down to issues.txt, an array of ~4K static &strs which is being inlined into the code it seems.
When I cut the file down to just a handful of items, compilation time reduces to 7 seconds 🙃

rust/src/tools/tidy/src/ui_tests.rs

Line 108 in 3d5528c

let issues_list =

cc @workingjubilee I guess?

@onur-ozkan onur-ozkan added A-testsuite Area: The testsuite used to check the correctness of rustc and removed needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Apr 1, 2024
@onur-ozkan
Copy link
Member

Can you check if #123339 reduces this slowness?

@matthiaskrgr
Copy link
Member Author

It does indeed.
Build completed successfully in 0:00:06

@onur-ozkan onur-ozkan mentioned this issue Apr 1, 2024
Merged
@onur-ozkan onur-ozkan self-assigned this Apr 1, 2024
@workingjubilee
Copy link
Member

Ah, but if people give the tests meaningful names, it will reduce the build time. 😌

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Apr 6, 2024
@matthiaskrgr
Rollup merge of rust-lang#123339 - onur-ozkan:optimize-tidy-check, r=…
534f20a 
…Mark-Simulacrum

optimize tidy check on `src/tools/tidy/src/issues.txt`

This change applies to the following:

- Handles `is_sorted` in the first iteration without needing a second.
- Fixes line sorting on `--bless`.
- Reads `issues.txt` as str rather than making it part of the source code.

Fixes rust-lang#123002
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Apr 6, 2024
@matthiaskrgr
Rollup merge of rust-lang#123339 - onur-ozkan:optimize-tidy-check, r=…
620d1bc 
…Mark-Simulacrum

optimize tidy check on `src/tools/tidy/src/issues.txt`

This change applies to the following:

- Handles `is_sorted` in the first iteration without needing a second.
- Fixes line sorting on `--bless`.
- Reads `issues.txt` as str rather than making it part of the source code.

Fixes rust-lang#123002
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Apr 6, 2024
@matthiaskrgr
Rollup merge of rust-lang#123339 - onur-ozkan:optimize-tidy-check, r=…
e27f5c4 
…Mark-Simulacrum

optimize tidy check on `src/tools/tidy/src/issues.txt`

This change applies to the following:

- Handles `is_sorted` in the first iteration without needing a second.
- Fixes line sorting on `--bless`.
- Reads `issues.txt` as str rather than making it part of the source code.

Fixes rust-lang#123002
@bors bors closed this as completed in 773fb88 Apr 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@onur-ozkan onur-ozkan

Labels
A-testsuite Area: The testsuite used to check the correctness of rustc T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

optimize tidy check on src/tools/tidy/src/issues.txt onur-ozkan/rust
4 participants
@matthiaskrgr @onur-ozkan @workingjubilee @rustbot