[vc] Investigate late attestations published on mainnet

[twoeths]

Describe the bug

• This is about the time we publish attestations compared to 1/3 of slot (so -2 means 1s after slot, +2 means 6s after slot
• On goerli it rarely happens

• However some mainnet nodes (v1.2.x) experienced this issue more

Expected behavior

Attestations should be published not later than 1/3 of slots

[twoeths]

This is because produceAttestationData being called after 4s (when vc hasn't received new chain head on time)

and produceAttestationData request time sometimes take >1s or even 5s

root cause is the I/O lag issue

[twoeths]

We could try http 1.1 keep-alive (or http 2, that's a longer plan) to reuse http connection

[philknows]

This is because produceAttestationData being called after 4s (when vc hasn't received new chain head on time)

and produceAttestationData request time sometimes take >1s or even 5s

root cause is the I/O lag issue

Referencing #4002

[twoeths]

I've tried http 1.1 keep-alive, it does not help while v1.3.0 shows some improvement, see #4915 (comment)

[dapplion]

Closing issues for specific incidents on old versions of Lodestar, please re-open if it happens with latest stable