Optimize UX loading +100 keystores on vc

[dapplion]

Describe the bug

After fixing #4166 now it takes a long time to load secure scrypt keystores.

Note that this UX improvement only affects whales and institutional players:

• Home stakers are likely to have < 10 keys
• Testnets + Kurtosis are run with either insecure scrypt keystores or mnemonics

Next steps:

• There's a bug currently where memory is not released after decrypting the keystores. Must investigate and ensure memory is released
• Instead of decrypting 100 keystores at once, decrypting 5-10 is fine, which provides a significant speed up. A good number would be to decrypt thread count at once
• Implement similar strategy to Prysm and Lighthouse where private keys are stored in a single keystore such that subsequent start-ups are fast

[philknows]

@wemeetagain @dapplion Do we still want to keep this issue open to address some of the ideal next steps listed in the original post? Unless I'm mistaken and these have actually been completed?

[wemeetagain]

We can reopen, but the last item is low priority, imo. The big win was @nazarhussain 's recent PR #5043

Last item is:

Instead of decrypting 100 keystores at once, decrypting 5-10 is fine, which provides a significant speed up. A good number would be to decrypt thread count at once

[nflaig]

A good number would be to decrypt thread count at once

in node v19.4.0 a new API was added to properly determine this, os.availableParallelism(), v20 will be active LTS in 8 months (2023-10-24) but initial release is already in April (2023-04-18).

[philknows]

This was brought up at the March 7, 2022 standup to prioritize the remaining item so institutional type node operators would be able to consider Lodestar in their setup. This is in reference to the tweet made by node operator RockLogic as a blocker to allow them to consider our validator client for their setup: https://twitter.com/Stefa2kEth/status/1628407293325807623?s=20

cc: @dapplion