-
Notifications
You must be signed in to change notification settings - Fork 12.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhance report-time option #64873
Enhance report-time option #64873
Conversation
b49f5e2
to
445e17e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall, this looks really good! If you think the unstable book makes more sense than the cargo book, that's fine with me. I just want to make sure there's documentation somewhere but it doesn't matter too much where.
cc @Centril in case they had any feedback. Otherwise I'll r+ tomorrow. |
src/libtest/lib.rs
Outdated
@@ -601,6 +802,19 @@ pub fn parse_opts(args: &[String]) -> Option<OptRes> { | |||
}; | |||
} | |||
|
|||
let report_time = unstable_optflag!(matches, allow_unstable, "report-time"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This part from let report_time
until else { None };
can be extracted into a function.
@@ -76,13 +77,54 @@ mod tests; | |||
const TEST_WARN_TIMEOUT_S: u64 = 60; | |||
const QUIET_MODE_MAX_COLUMN: usize = 100; // insert a '\n' after 100 tests in quiet mode | |||
|
|||
/// This small module contains constants used by `report-time` option. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The file is pretty large already (2kloc) so it would be good to add the new related changes into a dedicated file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree, but I think it will be better to not only move new changes into new file, but refactor the whole file and create several logically-consistent files (e.g. cli.rs
, time_options.rs
, benches.rs
, etc).
I'm going to do that in the next PR. Will it be OK?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm going to do that in the next PR. Will it be OK?
Sure, that sounds great!
How does this fit with e.g. https://github.com/altsysrq/proptest and |
@Centril regarding the 50ms/100ms threshold for unit test. I've used the values proposed in the previous PR, and in general I agree that those values won't suit everyones needs. As a follow-up, in the one of the next PRs I can make an additional section in |
Well, regarding the entry in the Unstable book. I have done it locally, but tidy error: The Unstable Book has a 'library feature' section 'libtest-report-time' which doesn't correspond to an unstable library feature
some tidy checks failed I've opened a tracking issue: #64888, however, this check still fails. Can somebody please guide me what exactly should I do here? |
59c8524
to
0cdee32
Compare
fn get_time_options( | ||
matches: &getopts::Matches, | ||
allow_unstable: bool) | ||
-> Option<OptPartRes<TestTimeOptions>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Talking about quite over-complicated return type: as I mentioned before, I want to split libtest
into several modules, so every module will be able to parse options of itself.
Thus, we will have a bunch of functions like get_XXX_options
which will return Option<OptPartRes<T>>
, and one master function in cli.rs
which will combain options parts from different modules.
However, as I can see, currently None
is returned only if help
argument was passed, so I guess I'll be able to simplify signature to be just OptPartRes<T>
.
☔ The latest upstream changes (presumably #64158) made this pull request unmergeable. Please resolve the merge conflicts. |
src/libtest/formatters/pretty.rs
Outdated
TrBench(ref bs) => { | ||
self.write_bench()?; | ||
self.write_plain(&format!(": {}\n", fmt_bench_samples(bs))) | ||
self.write_plain(&format!(": {}\n", fmt_bench_samples(bs)))?; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You added a \n
after the match, so you can remove this one.
src/libtest/lib.rs
Outdated
) | ||
.optflag( | ||
"", | ||
"ensure-test-time", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This parameter name could be more consistent with report-time
. Either
report-time
andensure-time
orreport-test-time
andensure-test-time
|
||
|
||
// If test is already failed (or allowed to fail), do not change the result. | ||
if result != TrOk { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you need to check result != TrOk && result != TrAllowedFail
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure. If test is allowed to fail, I think user will be confused if that test will anyway be failed in result.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure about that, either. Probably it doesn't matter because this feature will be removed, #46488.
Regarding merge conflict: I will resolve it soon (as well as review comment). However, I'm still waiting for any guidance for adding an entry to unstable book comment. |
src/libtest/lib.rs
Outdated
let time_options = match get_time_options(&matches, allow_unstable) { | ||
Some(Ok(val)) => val, | ||
Some(Err(e)) => return Some(Err(e)), | ||
x => panic!("Unexpected output from `get_time_options`: {:?}", x), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is x
always None
? Matching None
explicitly makes it clearer.
`RUST_TEST_TIME_UNIT`, `RUST_TEST_TIME_INTEGRATION` and | ||
`RUST_TEST_TIME_DOCTEST` environment variables. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there any message visible to the user explaining him that these environment variables contain the warn time and the critical time?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will be in the unstable book (and I will add a link to the corresponding page in that message) when somebody will answer my question.
src/libtest/lib.rs
Outdated
Threshold values for this option can be configured via | ||
`RUST_TEST_TIME_UNIT`, `RUST_TEST_TIME_INTEGRATION` and | ||
`RUST_TEST_TIME_DOCTEST` environment variables." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These environment variables take two numbers. Which number is the "test execution time limit"?
@Mark-Simulacrum do you know the correct way to resolve the tidy error in #64873 (comment)? |
I don't see any changes to the unstable book in this PR, which would be helpful -- I don't think we auto generate it? Initial suspicion is that you want underscores instead of dashes somewhere, though. |
0cdee32
to
d7423f1
Compare
Well, I've resolved an merge confict issue and (hopefully) all the review comments. r? @wesleywiser @Mark-Simulacrum I've committed an entry that I have added to the unstable book, so I will be really gratefull, if you'll tell me what I should do to resolve tidy error. |
The job Click to expand the log.
I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
The job Click to expand the log.
I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
Ah. The feature you've added is not a library feature (as usage does not imply |
@Centril @jakoschiko Are you OK with the current state of this PR? |
Code seems pretty clean from my end. :) |
@bors r+ |
📌 Commit 15f571b has been approved by |
The current state looks good. |
Enhance report-time option ## Short overview This PR is a follow-up to a previously closed #64714 PR. ## Changes introduced by this PR * `libtest` now retrieves the type of the test within `TestDesc` (available types are: `UnitTest`, `IntegrationTest`, `DocTest`, `Unknown`). * `--report-time` subcommand of the `libtest` now supports colored output (disabled by default). * Colorized output depends on the threshold values. Default values (proposed by @wesleywiser): - For unit-tests: 50ms warn/100ms critical, - For integration-tests: 500ms warn/1000ms critical, - For doctests: same as for integration tests, - For unknown tests: `TEST_WARN_TIMEOUT_S` warn/ `TEST_WARN_TIMEOUT_S * 2` critical (it will only applied single-threaded mode, because otherwise test will be interrupted after reaching `TEST_WARN_TIMEOUT_S`). - These values can be overrided by setting environment variables (since those thresholds are somewhat constant for every project, it's more flexible to use environment variables than command line arguments). * New optional flag `--ensure-test-time` for `libtest`. With this flag applied, exectuion time limit excesss will cause test failure. ## What have not been done There was a comment that it would be nice to have an entry in the Cargo book about it. However, changes introduced by this PR (and #64663 in which `report-time` flag was added) aren't related directly to `cargo`, it's more about `libtest` itself. I'm considering that [The Unstable Book](https://doc.rust-lang.org/unstable-book/) is more appropriate place, but not sure if I'm right (and if so, how exactly it should be described). As one possible option, this PR may be merged without denoting it in the documentation, and in the next PR adding support of this feature to the `cargo` itself, I'll add a note in the Cargo book. ## Scope of this PR Logical scope of this PR is `libtest` only. However, to get test types, I had to modify also `libsyntax_ext` and `librustdoc` for them to provide information about test type. ## Rationale Rationale for colored output was submitted in #64714 Providing the information about kind of test was also proposed in #64714, and as an additional benefit this information may be useful for the tools using `libtest` (e.g. `cargo`). Adding flag to treat time limits excess seems logical to me, so projects that do care about test execution time won't have to invent a wheel. ## Backward compatibility All the changes are completely backward compatible. ## Demo ![rustc_enhanced_time](https://user-images.githubusercontent.com/12111581/65818381-c04f6800-e219-11e9-9875-322463abe24f.gif) r? @wesleywiser
☀️ Test successful - checks-azure |
📣 Toolstate changed by #64873! Tested on commit 80b861b. 💔 clippy-driver on windows: test-pass → test-fail (cc @mcarton @oli-obk @Manishearth @flip1995 @yaahc @phansch @llogiq, @rust-lang/infra). |
Tested on commit rust-lang/rust@80b861b. Direct link to PR: <rust-lang/rust#64873> 💔 clippy-driver on windows: test-pass → test-fail (cc @mcarton @oli-obk @Manishearth @flip1995 @yaahc @phansch @llogiq, @rust-lang/infra). 💔 clippy-driver on linux: test-pass → test-fail (cc @mcarton @oli-obk @Manishearth @flip1995 @yaahc @phansch @llogiq, @rust-lang/infra). 💔 miri on windows: test-pass → test-fail (cc @oli-obk @eddyb @RalfJung, @rust-lang/infra). 💔 miri on linux: test-pass → test-fail (cc @oli-obk @eddyb @RalfJung, @rust-lang/infra).
`report_time` was renamed to `time_options` and is now an `Option`.
For `libtest` tests, this utilizes the unstable rustc feature added by rust-lang/rust#64873 to meter the execution time.
For `libtest` tests, this utilizes the unstable rustc feature added by rust-lang/rust#64873 to meter the execution time.
Short overview
This PR is a follow-up to a previously closed #64714 PR.
Changes introduced by this PR
libtest
now retrieves the type of the test withinTestDesc
(available types are:UnitTest
,IntegrationTest
,DocTest
,Unknown
).--report-time
subcommand of thelibtest
now supports colored output (disabled by default).TEST_WARN_TIMEOUT_S
warn/TEST_WARN_TIMEOUT_S * 2
critical (it will only applied single-threaded mode, because otherwise test will be interrupted after reachingTEST_WARN_TIMEOUT_S
).--ensure-test-time
forlibtest
. With this flag applied, exectuion time limit excesss will cause test failure.What have not been done
There was a comment that it would be nice to have an entry in the Cargo book about it.
However, changes introduced by this PR (and #64663 in which
report-time
flag was added) aren't related directly tocargo
, it's more aboutlibtest
itself.I'm considering that The Unstable Book is more appropriate place, but not sure if I'm right (and if so, how exactly it should be described).
As one possible option, this PR may be merged without denoting it in the documentation, and in the next PR adding support of this feature to the
cargo
itself, I'll add a note in the Cargo book.Scope of this PR
Logical scope of this PR is
libtest
only. However, to get test types, I had to modify alsolibsyntax_ext
andlibrustdoc
for them to provide information about test type.Rationale
Rationale for colored output was submitted in #64714
Providing the information about kind of test was also proposed in #64714, and as an additional benefit this information may be useful for the tools using
libtest
(e.g.cargo
).Adding flag to treat time limits excess seems logical to me, so projects that do care about test execution time won't have to invent a wheel.
Backward compatibility
All the changes are completely backward compatible.
Demo
r? @wesleywiser