Consider increasing default value for --max-old-space-size

[nflaig]

Problem description

Current default value (4096) is not sufficient for Holesky and causes node to crash during sync or unhealthy network conditions. Mainnet state is growing and we should consider increasing the default value before we run into the same issue there.

Solution description

Increase --max-old-space-size default value from 4096 to 6144.

Additional context

--max-old-space-size is set in Dockerfile and Lodestar shell script

lodestar/Dockerfile

Line 45 in 8afbc98

ENV NODE_OPTIONS=--max-old-space-size=4096

lodestar/lodestar

Line 7 in 8afbc98

node --trace-deprecation --max-old-space-size=4096 ./packages/cli/bin/lodestar.js "$@"

[twoeths]

We will need to reevaluate this value after n-historical state work #5968

[dapplion]

Can this value be set dynamically depending on the host's memory? Assume Lodestar is running on a host with 16GB or RAM, why limit ourselves to 4GB only?

[nflaig]

I noticed that node itself already sets it dynamically based on some host stats / configuration, default is 4GB on my machine

node -e "console.log(require('v8').getHeapStatistics().heap_size_limit / 1024 ** 3)"
4.046875

Could set it based on free memory (or total memory)

--max-old-space-size=$(expr $(echo "$(free -m)" | awk '/^Mem:/ {print $2}') - 512)

But I am not sure if that's good either as it is not adjusted dynamically during runtime, so it would depend on the startup order of EL / CL and other services running on the host.

I think it's important that it can be easily configured by operators as you might have to increase it during long periods of non-finality, which is quite easy on docker based deployments as a lot of those provide a environment variable for it. On dappnode, we also use a higher value for holesky at the moment which is configured on the package itself.