You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We should include in the set of github actions some benchmarks, for all target runtimes, to make sure we don't introduce accidental performance regressions. For example, #634 describes a large performance difference that came up from a change in compiled output that was completely unexpected for me when I made them.
Considering the complexity of optimal work distribution for the GPU runtime, I expect this to happen again.
It's important that we can quickly figure out these changes and find the problematic changes. Considering how fickle the performance can be, it can be very hard to understand where the regression comes from.
I propose to include in bend a standard set of benchmarking programs that we should run on every PR (and that each dev can run on their own machine as well). We need to decide which tools we want to use for that, but it should measure not only time, but number of interactions/interactions per second as well.
We need to make sure that for the Github actions they're run in an environment with stable performance. With the HVM benchmarks we used to have the problem of benchmakrs having unexpected low performance because something else was using the machine at the same time.
The text was updated successfully, but these errors were encountered:
We should include in the set of github actions some benchmarks, for all target runtimes, to make sure we don't introduce accidental performance regressions. For example, #634 describes a large performance difference that came up from a change in compiled output that was completely unexpected for me when I made them.
Considering the complexity of optimal work distribution for the GPU runtime, I expect this to happen again.
It's important that we can quickly figure out these changes and find the problematic changes. Considering how fickle the performance can be, it can be very hard to understand where the regression comes from.
I propose to include in bend a standard set of benchmarking programs that we should run on every PR (and that each dev can run on their own machine as well). We need to decide which tools we want to use for that, but it should measure not only time, but number of interactions/interactions per second as well.
We need to make sure that for the Github actions they're run in an environment with stable performance. With the HVM benchmarks we used to have the problem of benchmakrs having unexpected low performance because something else was using the machine at the same time.
The text was updated successfully, but these errors were encountered: