High regen queue job wait time

[twoeths]

Describe the bug

With the new gossip flow, the Job Wait Time of regen queue is almost 1s, this is likely one of the reason of I/O lag issue

right now the regen queue length is limited to 16 to accept new gossip work

Expected behavior

• This is from stable (v1.7): regen queue is only 1 or 0 and job wait time is way lower

Question

• We have a state cache of 96 items, why are there some attestations with block head not in that cache, is it worth to always regen them? maybe we should skip some of them, see

  lodestar/packages/beacon-node/src/chain/validation/attestation.ts

  Line 139 in 3ee1d8a

  // TODO (LH): Enforce a maximum skip distance for unaggregated attestations.

• Consider reducing REGEN_CAN_ACCEPT_WORK_THRESHOLD (right now it's 16) cc @dapplion

[twoeths]

when I zoom in, some Job Wait Time is > 4s which is too big

[dapplion]

@tuyennhv for this type of metric is best to capture the histogram to understand the real distribution of values through a longer timeframe. That would allow to answer the question: what % of regen jobs over a day take > 1s

[twoeths]

@dapplion it's about 25% of them take >1s

this issue happens randomly on stable mainnet node but more consistently on unstable mainnet node (could be because unstable likely validate all attestations with the new gossip)

[twoeths]

Sometimes getBlockSlotState() call of precomputeEpoch takes up to 3s

update: should investigate the long epoch transition in #5409

[twoeths]

Latest 1h chart in the last 24h in unstable mainnet node:

vs stable mainnet node