Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[mainnet] Epoch Transition time spiked #6063

Closed
twoeths opened this issue Oct 25, 2023 · 3 comments · Fixed by #6096
Closed

[mainnet] Epoch Transition time spiked #6063

twoeths opened this issue Oct 25, 2023 · 3 comments · Fixed by #6096
Assignees
@twoeths
Labels
meta-bug Issues that identify a bug and require a fix.

Comments

@twoeths
Copy link
Contributor

twoeths commented Oct 25, 2023

Describe the bug

On a mainnet node, epoch transition time spiked and never come back

Screenshot 2023-10-25 at 17 22 56

other nodes were restarted and this time got back to normal

Screenshot 2023-10-25 at 17 24 05

Expected behavior

Epoch transition on mainnet node should be consistently < 2s

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.11 and unstable
@twoeths twoeths added the meta-bug Issues that identify a bug and require a fix. label Oct 25, 2023
@twoeths twoeths mentioned this issue Oct 26, 2023
Merged
@twoeths
Copy link
Contributor Author

twoeths commented Oct 31, 2023

this also happens on stable 1k validator node where rss spiked (note that heap memory slowly increased) which caused gc time spiked and epoch transition time spiked

Screenshot 2023-10-31 at 20 51 41 Screenshot 2023-10-31 at 20 51 50 Screenshot 2023-10-31 at 20 52 06

@twoeths twoeths self-assigned this Nov 5, 2023
@philknows philknows mentioned this issue Nov 7, 2023
Open
This was referenced Nov 8, 2023
Merged
Merged
@twoeths
Copy link
Contributor Author

twoeths commented Nov 8, 2023

Some more observation on the issue:

  • It only happens with nodes that have validators
  • It tends to happen when we have a wasted epoch transition, we populated 3 checkpoint states per epoch at some points
  • large_object in V8 memory came up and never come back

this links to the finding in #6096

@twoeths twoeths reopened this Nov 9, 2023
This was referenced Nov 10, 2023
Merged
Open
@twoeths twoeths mentioned this issue Nov 30, 2023
Merged
@twoeths
Copy link
Contributor Author

twoeths commented Jul 25, 2024

haven't seen this issue for a while and code changed a lot, closing for now

@twoeths twoeths closed this as completed Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@twoeths twoeths

Labels
meta-bug Issues that identify a bug and require a fix.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: avoid leaky event handler in waitForCheckpointState ChainSafe/lodestar
1 participant
@twoeths