Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure to prune to target peers #5198

Closed
twoeths opened this issue Feb 23, 2023 · 2 comments · Fixed by #5217
Closed

Ensure to prune to target peers #5198

twoeths opened this issue Feb 23, 2023 · 2 comments · Fixed by #5217
Assignees
Labels
prio-high Resolve issues as soon as possible. scope-networking All issues related to networking, gossip, and libp2p.

Comments

@twoeths
Copy link
Contributor

twoeths commented Feb 23, 2023

Describe the bug

In rare case where all peers subscribe to at least 1 subnet, but they are not grouped to 1 subnet, and all performs well, lodestar doesn't prune peers to target peers. When that happens, we see peers stay >= 55 peers like below:

Screen Shot 2023-02-23 at 10 05 58

this does not allow new connections (implemented by libp2p). in worse case scenario, we don't have enough subnet peers (from same node like the above)

Screen Shot 2023-02-23 at 10 09 29

and it causes a lot of missed attestations (since this node has 1k validators on goerli and it's supposed to subscribe to all subnets)

Expected behavior

Make sure we always prune to target peers (50) to allow more better peers to come, like below:

Screen Shot 2023-02-23 at 10 11 41

@twoeths twoeths self-assigned this Feb 23, 2023
@twoeths
Copy link
Contributor Author

twoeths commented Feb 23, 2023

Adding no long lived subnet peers metric which is used to find this issue

Screen Shot 2023-02-23 at 10 29 51

Requested peer to disconnect metric is like

Screen Shot 2023-02-23 at 10 32 17

@philknows philknows added prio-high Resolve issues as soon as possible. scope-networking All issues related to networking, gossip, and libp2p. labels Feb 23, 2023
@twoeths
Copy link
Contributor Author

twoeths commented Feb 27, 2023

when this error happens, it leads to InsufficientPeers issue at validator side

grep -e "PublishError.InsufficientPeers" -rn validator-2023-02-23.log | wc -l
20070

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
prio-high Resolve issues as soon as possible. scope-networking All issues related to networking, gossip, and libp2p.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants