ci: Add benchmark performance comparisons #232
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jsimnz
added
area/testing
shahzadlone
force-pushed
the
lone/ci/benchmark-comparisons
branch
from
64967dd
to
6f50ebc
Compare
shahzadlone
commented
Mar 18, 2022
| github.event_name == 'pull_request' && | ||
| github.base_ref == 'develop' | ||
| run: > | ||
| make deps:bench && |
There was a problem hiding this comment.
This make deps:bench step will be removed once I modify the AMI image on AWS that is used to create the EC2 instance to contain this dependency from the get go.
shahzadlone
added
the
ci/build
It might be nice to have them run when the target is main too, gives a nice summary of the release cycle's impact on users. If it is not a quick thing to add this can added some other time perhaps EDIT: it looks like this would be a quick or check on two lines in the yaml, but I might be wrong - would suggest doing now |
AndrewSisley
approved these changes
Mar 18, 2022
shahzadlone
added a commit
to shahzadlone/defradb
that referenced
this pull request
Feb 23, 2024
* Implement benchmark comparisons using the last successfully uploaded benchmark report on the develop branch * See if GOPATH helps find the downloaded binary of benchstat * squeeze in one line to see if works now. * Add `-alpha 1.1` to benchstat to trick it to display the deltas
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DESCRIPTION
This PR modifies the CI flow as follows:
action/no-benchmarklabel is present, now if the PR is going into a branch that is NOTdevelopbranch (base reference !=develop) then the benchmarks will not run.develop) and the benchmarks are supposed to run, then CI will go fetch the latest artifact from the most recent build ondevelopthat passes thelint-then-benchmark.ymlworkflow (ensures artifact was uploaded) and run the benchmark comparisons using benchstat.ISSUE
Closes #212
Rough Notes On Tools Explored:
benchstat: is really simple and we can use built in github action cache to store the previous bench stats (rather than their way) and just run the comparisons and see the results through the CI run workflow (not fun to look at).github-action-benchmarkandgobenchdata: both are very similar but the github action benchmark gives you ability to have the benchmarks publish to lets say thedocsfolder inside themasterbranch. However output of gobench seems much cleaner (one downside is that we can't really or rather should not run this on PR because it will modify a branch).gobencher: I enabled this github app to generate PR results the tackle the problem of not being able to rungobenchdataorgithub-action-benchmarkon PRs. This creates a very nice comparison on the PR check itself. One down side is that we can't control this with labels