tests: remove flaky Chrome launch from unit-cli #12359
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the spirit of improving flaky tests,
unit-cli
has loooong had a flake based on a Chrome launch sometimes timing out. You know the one:(for example: https://github.com/GoogleChrome/lighthouse/runs/2249363555?check_suite_focus=true)
The issue is that this is more or less a smoke test without any of the functionality we've built into smokehouse to handle dealing with launching and testing with Chrome in a CI environment.
Looking way back, the original point of the test (from #2602) was to make sure that Lighthouse can run when
lighthouse-cli
is accessed directly as a node module, since our smoke tests load Lighthouse via CLI. Very similar to the smoke tests, but we have no other test ensuring we don't break this particular use case. Since then, the assertions have grown to check a few other aspects of the run.However the overlap between this and smokehouse's execution path is almost total (they both pass through
runLighthouse
inrun.js
), so it seems sufficient for this test to run lighthouse with--auditMode
and the sample artifacts to make sure it's runnable, and trust smokehouse to verifyrun.js
is exercising the gathering steps.After this I believe there's no other Chrome launch in
unit-cli
, so it should be stable going forward.