[improve][broker] Use shrink map for trackerCache #19534
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: xiaolongran <[email protected]>
wolfstudy
requested review from
Jason918,
eolivelli,
codelipenghui and
congbobo184
|
@wolfstudy Please add the following content to your PR description and select a checkbox: |
github-actions
bot
added
doc-not-needed
Signed-off-by: xiaolongran <[email protected]>
Jason918
reviewed
Feb 16, 2023
|
|
||
| public class InMemoryRedeliveryTracker implements RedeliveryTracker { | ||
|
|
||
| private ConcurrentLongLongPairHashMap trackerCache = new ConcurrentLongLongPairHashMap(256, 1); | ||
| private ConcurrentLongLongPairHashMap trackerCache = ConcurrentLongLongPairHashMap.newBuilder() | ||
| .concurrencyLevel(2) |
There was a problem hiding this comment.
Any reason to change the 256,1 to 128,2 ?
There was a problem hiding this comment.
Fixed done, PTAL again
Signed-off-by: xiaolongran <[email protected]>
lhotari
approved these changes
Feb 16, 2023
|
/pulsarbot rerun-failure-checks |
Technoboy-
approved these changes
Feb 16, 2023
liangyepianzhou
pushed a commit
that referenced
this pull request
Feb 25, 2023
Signed-off-by: xiaolongran <[email protected]> (cherry picked from commit c0f89dc)
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Feb 28, 2023
Signed-off-by: xiaolongran <[email protected]> (cherry picked from commit c0f89dc) (cherry picked from commit 0f455e6)
coderzc
pushed a commit
that referenced
this pull request
Mar 1, 2023
Signed-off-by: xiaolongran <[email protected]> (cherry picked from commit c0f89dc)
Technoboy-
pushed a commit
that referenced
this pull request
Mar 6, 2023
Signed-off-by: xiaolongran <[email protected]>
Annavar-satish
pushed a commit
to pandio-com/pulsar
that referenced
this pull request
Mar 6, 2023
Signed-off-by: xiaolongran <[email protected]> (cherry picked from commit c0f89dc)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Currently, the
trackerCacheuses a ConcurrentLongLongPairHashMap that does not shrink, causingtrackerCacheto occupy a large amount of memory and has no way to release it.By monitoring the market through the JVM, we can see that during that time period, Young GC and Full GC have been frequently triggered, resulting in a large delay in Young GC and Full GC, which in turn affects production and consumption.
But what is interesting is that we can see that after the Young GC and Full GC are triggered many times, the memory in the heap is not reclaimed, so the Young GC and Full GC are triggered frequently.
Therefore, based on the above background, we made a memory dump of the Broker node and found that the set of trackerCache occupies most of the memory and this part of memory uses a non-shrink map, which will never be released. For details, refer to the following dump information:
Modifications
Use shrink map for trackerCache
docdoc-requireddoc-not-neededdoc-complete