Performance Tests Optimization: Measure impact of changes #45175
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dmsnell
added
the
Exclude from Metrics
|
Size Change: 0 B Total Size: 1.3 MB ℹ️ View Unchanged
|
dmsnell
force-pushed
the
perf-tests/without-changes
branch
from
465b5ac
to
14b8ffe
Compare
dmsnell
force-pushed
the
perf-tests/without-changes
branch
3 times, most recently
from
4cd7f22
to
a538b7b
Compare
dmsnell
force-pushed
the
perf-tests/without-changes
branch
from
a538b7b
to
827b24f
Compare
dmsnell
force-pushed
the
perf-tests/without-changes
branch
from
827b24f
to
6118e60
Compare
|
Closing for now since active work on the performance tests performance is on hold. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
This branch is intended to follow
trunkwithout the set of changes meant improve the runtime performance of the Performance Test CI workflow.trunkas of this test is 20.6mintrunkand 76 test runs in this branch (there are more intrunkbecause the tests finish faster and I'm able to get more data in the same amount of time)trunkis 2.6min vs. without the changes at 7.5min, so not only has the average test time reduced but also the extrema for the test runs has also reduced and the runtimes are now more consistent.Results
refspecs#45780In order to eliminate some bias introduced by variations in test run durations over time these numbers reflect only the 30 most recent test runs.
Nov. 1
I'm rebasing all branches and am going to let them re-gather data to see if any clearer patterns emerge. Will be ignoring all test run data before Nov. 1 to see if there were effects due to getting too far behind
trunk.Why
By comparing the runtimes in this branch against those in
trunkwe can measure the impact of the overall optimization project while deploying improvements incrementally. Running this over time will not only eliminate sampling bias and noise, but it will more importantly help us to eliminate the impact of other changes on the performance test suite which might be conflated with changes we're making to the suite itself (e.g. if someone adds a new set of performance tests to the suite which would slow them down).