Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark baseline check doesn't respect threshold-tolerances when checking baseline against static thresholds #275

Open
MahdiBM opened this issue Sep 18, 2024 · 0 comments · May be fixed by #276

Comments

@MahdiBM
Copy link
Contributor

MahdiBM commented Sep 18, 2024

benchmark baseline check doesn't respect threshold-tolerances when checking already-acquired baseline which is on disk, against static thresholds .json files.
Apparently it's because the benchmarks are not loaded so the tool falls back to the strict threshold-tolerances.

This makes our benchmarks fail even though we have less than 1% and 100ms of fluctuations.

@MahdiBM MahdiBM linked a pull request Sep 18, 2024 that will close this issue
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

1 participant