Hermes Engine takes 2.5 to 19 times longer to process promises than Chakra

[tmikov]

Linking to microsoft#92 , so it can be discussed here.

[tmikov]

@vmoroz @AlexLablaiksSAP would it be possible to obtain a non-ReactNative repro?

[AlexLablaiksSAP]

Sure, I haven't played too much with Hermes outside of React Native, so I may need a little more context of what is most helpful for your team though.

Is there a template or sample app you would like me to use? If not:

• Is a reproduction via a command line or GUI preferred?
• Is there a specific Operating System your team will run through the reproduction steps on?
• Does this request include comparing the performance to Chakra (as the original steps and repository does)?

Thanks for the clarification!

[neildhar]

@AlexLablaiksSAP

• A repro using the Hermes CLI tool would be great.
• macOS or Linux would be the most convenient.
• Once you have a standalone repro, it should be easy to run in different engines (including Chakra). It would certainly be interesting if you observe significant differences in relative performance across platforms.

[AlexLablaiksSAP]

OK, so I've gone ahead and created hermes-promise-cli, and the results were surprising. Hermes on the command line plows through the tests:

 LOG  populated 10000 object cells in 19 ms
 LOG  formatted 10000 object cells in 195 ms via flat promises.
 LOG  formatted 10000 object cells in 195 ms via flat promises.
 LOG  formatted 10000 object cells in 192 ms via flat promises.
 LOG  formatted 10000 object cells in 194 ms via flat promises.
 LOG  formatted 10000 object cells in 194 ms via flat promises.
 LOG  populated 10000 object cells in 3 ms
 LOG  formatted 10000 object cells in 373 ms via nested promises.
 LOG  formatted 10000 object cells in 371 ms via nested promises.
 LOG  formatted 10000 object cells in 381 ms via nested promises.
 LOG  formatted 10000 object cells in 376 ms via nested promises.
 LOG  formatted 10000 object cells in 394 ms via nested promises.

While ChakraCore struggles to get through them on the command line:

populated 10000 object cells in 7 ms
formatted 10000 object cells in 13756 ms via flat promises.
formatted 10000 object cells in 13630 ms via flat promises.
formatted 10000 object cells in 13470 ms via flat promises.
formatted 10000 object cells in 13064 ms via flat promises.
formatted 10000 object cells in 12506 ms via flat promises.
populated 10000 object cells in 90 ms
formatted 10000 object cells in 15349 ms via nested promises.
formatted 10000 object cells in 16323 ms via nested promises.
formatted 10000 object cells in 15996 ms via nested promises.
formatted 10000 object cells in 15206 ms via nested promises.
formatted 10000 object cells in 15165 ms via nested promises.

However, I've reconfigured the React Native project hermes-promise-test to use hermes-promise-cli as a submodule and then proceeded to test it against Android as well and it looks like Android struggles just as much as Windows does.

So, it looks to me that something is happening at the react-native level that is affecting the behavior of the JavaScript engines. However, I'm not sure if this is a Metro, Babel, react-native command, or react-native environment issue.

Note: The cli project uses parcel to create an executable bundle, which primarily transforms the classes while leaving most syntax alone.

Any guidance on next steps for investigation, which repository might be the cause of the issue, or hidden configuration settings during React Native bundling?

[neildhar]

@AlexLablaiksSAP thank you for sharing your results. One immediate suspect is overhead in JSI and populating RN's task queue. The performance of Hermes' JSI layer has improved dramatically recently though.

I noticed that you're on RN 0.68, are you able to try this out with the latest version of RN? If JSI overhead in Hermes is the culprit, it is likely you will see better performance.

[AlexLablaiksSAP]

@neildhar I originally opened hermes-promise-test against 0.68, but it is on 0.71 now (for Windows Client Compatibility), see main's package.json of hermes-promise-test.

I have gone ahead and pulled in 0.72 and regenerated a fresh project and ran it on Android, while it has halved the time to around 5080 ms, it still is behind JavaScriptCore's 524 ms.

[neildhar]

Interesting. Is the comparison with JSC done with a debug or release build of the RN app?

[evelant]

@neildhar Maybe related but I was having performance problems in my app on Android so I tested it with JSC and v8. JSC is approx 4x faster than hermes. V8 is 6x or more faster.

I think these problems might have started around the release of react-native 0.70. As of now hermes performance is so bad it's unusable in production. This is a tough situation since JSC and v8 are now "second class" and seem to have bit-rotted somewhat causing different issues on latest react-native.

My app does make heavy use of some JSI libraries (react-native-mmkv and react-native-skia) so that's a possible hot spot. My app also heavily uses mobx which means it uses a lot of proxies, another possible problem spot.

Sorry this information is somewhat vague. I have no idea what's causing the terrible performance on hermes at the moment. I also have little idea about how I can debug it.

[tmikov]

@evelant when you say 4x or 6x faster, can you be a little more precise? What exactly is faster? Startup time? Update rate? Do you have a benchmark demonstrating the performance difference?

[evelant]

@tmikov I wish I had a benchmark but at the moment I don't know exactly what's causing the problem. Startup time and overall performance appears to be approx 5x worse with hermes compared to v8 in my app. My best guess so far is that JSI libraries or proxies could be the culprits but that's a weak guess. Sorry I don't have more useful information at the moment but I'll let you know if I uncover some.

[tmikov]

@evelant if you are experiencing 5 time longer startup with Hermes, that is a strong indicator of a systemic problem like running from source instead of bytecode.

It is extremely unlikely that Hermes execution itself is 5 times slower on startup. Even if we ignore that Hermes has significantly faster load times than other engines (it is smaller, there is no parsing and compilation of JS, input is memory mapped and lazily loaded, etc), it also has competitive interpreter performance. Startup time is not enough for JIT warmup where V8 and JSC JITs would start helping (if it was, then by definition you would be doing too much on startup).

My first guess would be, you are running from source instead of bytecode.

[evelant]

I double checked and I am definitely not running from source.

I think I must be tripping over a performance corner case such as #811 #930 #1008 or the original issue here.

Testing again I can see that startup performance (time until my code begins executing) is somewhat similar between all engines (hermes is still slowest but not by much). Runtime performance however is much worse with hermes than v8 or JSC. Unfortunately it's tough to figure out the root cause.

[AlexLablaiksSAP]

Interesting. Is the comparison with JSC done with a debug or release build of the RN app?

I've gone ahead and upgraded to 0.72.6 and re-ran the app at hermes-promise-test. Both results are run in Release mode with 7.4 seconds vs 1.1 seconds. I have noticed that there has recently been an improvement, but it's still around 6-7x the time for both Android and Windows. For convenience, here are the screenshots of both engines on Android, Windows is fairly similar at the moment.

Any idea on what might be happening in the 0.72.6 React Native environment that makes the results so different from the plain hermes usage? I have been speculating that maybe it is a hidden polyfill, but this doesn't appear to be the case.

I'm suspecting it might be best to open an issue against the React Native repository directly.

[AlexLablaiksSAP]

I've gone ahead and upgraded and re-ran the hermes-promise-test against React Native 0.75.1 on an Android Emulator. See hermes-promise-test-android.xlsx

JavaScriptCore outperforms Hermes with a flat amount of 10,000 promises as well as the scenario of 1,000 nested promises. The ratios vary from Hermes taking 1.28x longer to 8x longer at the peak (10K nested). Feel free to pull the aforementioned repository and test.

Note: Our real world scenario is where we have an executed OData query result where each entity received has their fields formatted (the nested promise scenario).

[tmikov]

We have landed speedups to Array.prototype.indexOf(), which may fix this issue: 063293b .

However the fix is in our static_h branch, so trying it requires some gymnastics.