Validator performance improvements

[dapplion]

General epic to track all issues to improve validator performance and profitability:

Root: #3527

On-going issues

• Investigate why there are so few mesh peers per attestation subnet #3555
• PeerManager: prioritize peers to disconnect #3832
• Node receives late gossip block #3907
• Validator does not submit attestations right after chain head comes #3911
• Timeout publishing unaggregated attestation #3922
• PeerManager failed to connect to subnet peers #3940
• Investigate Wrong Head Ratio metric #3985
• Review AggregateAndProof op pool #4028
• [PROD] Validator votes for incorrect head #4009

Gossipsub issues

• New parameter: pruneBackoff and unsubcribeBackoff js-libp2p-gossipsub#249

Additional metrics / tools

• Track additional external validators in lodestar #3974

[dapplion]

Notes from https://discord.com/channels/613988663034118151/812268053718237196/935076545519845396

• more aggressive outgoing connections to increase the number of peers gossisub peers for every subnet (Tune getLowSubnets status-im/nimbus-eth2#3241)
• shorter backoff so that resubscribing to an aggregation subnet works (Customizable gossipsub backoff on unsubscribe vacp2p/nim-libp2p#665) - this is important for people with 20+ validators on a single node
• disallow concurrent connections to the same peer (Tune getLowSubnets status-im/nimbus-eth2#3241) - this prevents junk peers from taking up spots in the connection table and fixes some ordering bugs with subscription / unsubscription in the mesh

[twoeths]

note from Nimbus PR 3241 above:

• find subnet peer with outbound peers <D_OUT peers
• or if we subscribe to a subnet, and < D_LOW mesh peers
• or subnets with topic peers < D_LOW(even if we don't subscribe to that subnet)
• or subnets with topic peers < D_HIGH

Given GossipsubDout = 2 and we have too many outbound peers out of 50 (target) peers, we don't need that logic for now. It's nice to have other logic to find subnet peers, but we also have equivalent logic in lodestar so will consider this if we missed attestations.

With v0.36.0 and a mainnet node of 8 connected validators, topic peers look nice so no motivation to work on that right now

[dapplion]

Almost all points covered, closing in favor of future more granular issues for specific improvements