-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Benchmarking results are unstable for native target due to object pinning done by the Blackhole #114
Comments
Worth mentioning that in general, it's not a viable option to perform some sort of consumption under a never-executed branch as after inlining some sophisticated compiler optimizations (like GraalVM's control-flow sensitive partial escape analysis) may move actual work performed by the benchmark to that branch, but it's very unlikely to be an issue with KMP (at least for now). |
Created a reproducer that can catch the problem with unstable results: https://github.com/fzhinkin/kotlinx-benchmark-native-blackhole-reproducer/blob/main/build.gradle.kts#L110 |
Re-implemented native blackhole without using object pinning as it affects performance and leads to unstable results as each time GC has to spend more and more time scanning all pinned values. Instead, primitive values consumption is implemented as a comparison of the value for equality with two fields and publishing the value in case when comparison succeeds. The values themselves are never the same and one of the fields is volatile, thus the condition is always false and it could not be omitted because of volatility. That should prevent both dead code elimination and movement of the code computing the consumed value into an effectively unreachable branch. For the objects, identifyHashCode is used to obtain an int-value that is then passed into a regular consumption routine. That function is an intrinsic that simply gets an address of the object, so it has no performance impact, yet it requires an object. Fixes #114
…e.maven.plugins-maven-surefire-plugin-3.0.0 build(deps): bump maven-surefire-plugin from 3.0.0-M9 to 3.0.0
Benchmarking results on native platforms are unstable and measured performance degrades from iteration to iteration.
Issue
While running some benchmarks for the native target I spotted the problem: performance degrades from iteration to iteration.
The root cause is how the Blackhole is implemented for native: its
consume
methods pins a value: https://github.com/Kotlin/kotlinx-benchmark/blob/master/runtime/nativeMain/src/kotlinx/benchmark/NativeBlackhole.kt#L7References pinned by the blackhole are never unpinned and that exposes additional work for native GC as every pinned reference is a GC root.
The issue could be reproduced with the KMP example from the repository, but I added an additional configuration (204c223) to make the issue more obvious:
As you can see, measured time drifts significantly, and it degrades with each iteration.
Here's a flame graph built from dtrace profiling results:
Suggested fix
To avoid the issue NativeBlackhole may use the same approach as the JMH's blackhole and pin objects conditionally with a condition that is always false, like:
Here's a commit with a suggested workaround: fzhinkin@439e2ac
Note that actual pinning is not necessary and we can save the
consume
's argument to some public field instead.With that change benchmarks execution results are stable:
The text was updated successfully, but these errors were encountered: