Fix Gossip excessive memory consumption by SeenCache #300
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
SeenCache
aims to effectively filter duplicate messages for a longer period (2 minutes by default) thanmCache
stores (5 seconds by default). TheSeenCache
implementations shouldn't retain references to received messages' content since it could be significant memory consumption in case of heavy gossip traffic:The secondary purpose of the
SeenCache
is to cachemessageId
(which calculation could be resource consuming) of seen messages based of a fast message id (seeSeenCache.getSeenMessageCached()
method andFastIdSeenCache
implementation)To address the memory issue the
SeenCache
implementations should rely on message ids (either slow or fast or both) and don't keep thePubsubMessage
references.To preserve the secondary purpose the
FastIdSeenCache.getSeenMessageCached()
wraps thePubsubMessage
with a wrapper instance which is supplied with the cachedmessageId
when available.Unfortunately there are no reliable means to create unit tests for memory leak/excessive usage cases