Fix Memory Leak In New Timecache Implementations #528
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vyzo
approved these changes
Mar 15, 2023
This was referenced Mar 15, 2023
yhassanzadeh13
added a commit
to yhassanzadeh13/go-libp2p-pubsub
that referenced
this pull request
Jul 3, 2023
* feat: expire messages from the cache based on last seen time (libp2p#513) * feat: expire messages from the cache based on last seen time * chore: minor renaming * fix: messages should not be found after expiration * chore: editorial * fix: use new time cache strategy consistently * fix: default to old time cache and add todo for background gc * chore: update to go-libp2p v0.25 (libp2p#517) * Update to go-libp2p v0.25 * Use go 1.19 * chore: update go version and dependencies (libp2p#516) * fix(timecache): remove panic in first seen cache on Add (libp2p#522) * Refactor timecache implementations (libp2p#523) * reimplement timecache for sane and performant behaviour * remove seenMessagesMx, take advantage of new tc api * fix timecache tests * fix typo * store expiry, don't make life difficult * refactor common background sweep procedure for both impls * add godocs to TimeCache * Default validator support (libp2p#525) * add default validator support * add an implementation for basic seqno as nonce validation * missing return * the nonce belongs to the origin peer * add note about rust predicament * add seqno validator tests * minor test tweak, ensure at least 1ms before replay * Fix Memory Leak In New Timecache Implementations (libp2p#528) * fix bug * add for last seen cache * chore: Update .github/workflows/stale.yml [skip ci] * chore: Update .github/workflows/stale.yml [skip ci] * upgrades libp2p version * upgrades libp2p version --------- Co-authored-by: Mohsin Zaidi <[email protected]> Co-authored-by: Marco Munizaga <[email protected]> Co-authored-by: RichΛrd <[email protected]> Co-authored-by: Hlib Kanunnikov <[email protected]> Co-authored-by: vyzo <[email protected]> Co-authored-by: Nishant Das <[email protected]> Co-authored-by: GitHub <[email protected]>
yhassanzadeh13
added a commit
to yhassanzadeh13/go-libp2p-pubsub
that referenced
this pull request
Feb 20, 2024
* feat: expire messages from the cache based on last seen time (libp2p#513) * feat: expire messages from the cache based on last seen time * chore: minor renaming * fix: messages should not be found after expiration * chore: editorial * fix: use new time cache strategy consistently * fix: default to old time cache and add todo for background gc * chore: update to go-libp2p v0.25 (libp2p#517) * Update to go-libp2p v0.25 * Use go 1.19 * chore: update go version and dependencies (libp2p#516) * fix(timecache): remove panic in first seen cache on Add (libp2p#522) * Refactor timecache implementations (libp2p#523) * reimplement timecache for sane and performant behaviour * remove seenMessagesMx, take advantage of new tc api * fix timecache tests * fix typo * store expiry, don't make life difficult * refactor common background sweep procedure for both impls * add godocs to TimeCache * Default validator support (libp2p#525) * add default validator support * add an implementation for basic seqno as nonce validation * missing return * the nonce belongs to the origin peer * add note about rust predicament * add seqno validator tests * minor test tweak, ensure at least 1ms before replay * Fix Memory Leak In New Timecache Implementations (libp2p#528) * fix bug * add for last seen cache * chore: Update .github/workflows/stale.yml [skip ci] * chore: Update .github/workflows/stale.yml [skip ci] * bump golang.org/x/net from 0.4.0 to 0.7.0 (libp2p#520) Bumps [golang.org/x/net](https://github.com/golang/net) from 0.4.0 to 0.7.0. - [Release notes](https://github.com/golang/net/releases) - [Commits](golang/net@v0.4.0...v0.7.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: topicscore params can't be set for dynamically subscribed topic (libp2p#540) * fix: topicscore params can't be set for a topic subscribed after gossipsub is initialized * chore:address review comments * Revert "fix: topicscore params can't be set for dynamically subscribed topic (libp2p#540)" (libp2p#541) This reverts commit aa5fd79. * remove usage of deprecated peerid.Pretty method (libp2p#542) * chore: update go-libp2p to v0.32 (libp2p#548) * chore: Update .github/workflows/stale.yml [skip ci] * make tidy --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Mohsin Zaidi <[email protected]> Co-authored-by: Marco Munizaga <[email protected]> Co-authored-by: RichΛrd <[email protected]> Co-authored-by: Hlib Kanunnikov <[email protected]> Co-authored-by: vyzo <[email protected]> Co-authored-by: Nishant Das <[email protected]> Co-authored-by: GitHub <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Prem Chaitanya Prathi <[email protected]> Co-authored-by: Sukun <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In our latest release of prysm, we updated it to use version v0.9.2 of
go-libp2p-pubsub. However, immediately after a few hourswe received multiple reports of memory leaks with prysm and prysm nodes constantly crashing with an OOM. After fetching a few profiles, this is what we found:
The majority of heap usage was being consumed by the first seen cache. After digging into it and the PR which introduced it
#523 , it was found out that the background routine was not clearing our expired entries from the cache. This is because it used
NewTimerinstead ofNewTickerwhen performing the sweep. So this would only fire an event once into the channel and fire no more events after.This PR introduces the fix along with fixing the test so that this particular case is correctly checked for.