Implement regen metrics

[dapplion]

• Cache hits vs cache miss
• Cache frequency usage, last usage per item

[dapplion]

@dapplion I think we should close this in favor of suggestion above:
add metrics to regen module

From #2285 (comment)

[g11tech]

@dapplion please assign

[dapplion]

@dapplion please assign

If you need help to design the metrics on this component reach out to me or Cayman. Be aware that the regen logic is quite complex with many entrypoints. Doing the metrics right will be much harder than the fork choice and require a good understatement of it mechanics.

[g11tech]

@dapplion thanks, will try to figure it out while I implement other simpler metrics. Will circle back on this once I get a priliminary understanding. 👍

[g11tech]

ok, i think i have a preliminary understanding of the regen module basis which I propose following metrics structure:

regenStateCacheLookupTotal: gauge<entrypointFn,callingModule>
regenStateCacheLoopkupHits: gauge<entrypointFn,callingModule>

regenCPStateCacheLookupTotal: gauge<entrypointFn,callingModule>
regenCPStateCacheLookupHits: gauge<entrypointFn,callingModule>

where entrypointFn label in ['getPreState','getCheckpointState','getState','getBlockSlotState']
callingModule label in ['validateGossipBlock','validateGossipAttestation','validateGossipAggregateAndProof','validateGossipVoluntaryExit','produceBlock','produceAttestationData', 'getProposerDuties', 'getAttesterDuties', 'getSyncCommitteeDuties',', 'onFinalized']

This will give a good slicing, dicing and rollups to figure out and debut what is happening.
grafana dashboard panel, can plot them up as stacked, and a module wise sum by (callingModule) graphs can be plotted.

@dapplion @wemeetagain let me know your thoughts on the same.

[wemeetagain]

I thinkthe lookup / hits is a good start.

Here's how I see the regen module, there's a few different loads in the regen module that may happen, from best to worst:

• the requested state is in the cache, no further processing required
• the requested state at a prior slot is in the cache, slots must be dialed forward, possibly hitting an epoch transition
• a prior state is in the cache, blocks must be replayed to the requested state
• a prior state is in the cache, blocks must be replayed to the requested state, then slots must be dialed forward

And then there's the error case, when the state can't be regenerated for whatever reason.

I think some other things I'd like to know, where metrics could help:

• how often are we needing to reprocess slots? epochs? blocks?
  • how many blocks? (not very actionable, but maybe interesting)
• how long do regen tasks take?
• how often are there regen errors?