Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmark runs to the github actions #704

Open
developedby opened this issue Sep 4, 2024 · 0 comments
Open

Add benchmark runs to the github actions #704

developedby opened this issue Sep 4, 2024 · 0 comments
Labels
enhancement New feature or request tests

Comments

@developedby
Copy link
Member

We should include in the set of github actions some benchmarks, for all target runtimes, to make sure we don't introduce accidental performance regressions. For example, #634 describes a large performance difference that came up from a change in compiled output that was completely unexpected for me when I made them.

Considering the complexity of optimal work distribution for the GPU runtime, I expect this to happen again.

It's important that we can quickly figure out these changes and find the problematic changes. Considering how fickle the performance can be, it can be very hard to understand where the regression comes from.

I propose to include in bend a standard set of benchmarking programs that we should run on every PR (and that each dev can run on their own machine as well). We need to decide which tools we want to use for that, but it should measure not only time, but number of interactions/interactions per second as well.

We need to make sure that for the Github actions they're run in an environment with stable performance. With the HVM benchmarks we used to have the problem of benchmakrs having unexpected low performance because something else was using the machine at the same time.

@developedby developedby added enhancement New feature or request tests labels Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request tests
Projects
None yet
Development

No branches or pull requests

1 participant