fix: avoid excessive memory allocations #2389
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
harnash
force-pushed
the
avoid_excessive_memory_allocations
branch
from
49cf236
to
c55e16d
Compare
aeneasr
requested changes
Apr 11, 2022
| cache, err := ristretto.NewCache(&ristretto.Config{ | ||
| NumCounters: 10 * 10000, | ||
| MaxCost: 60 * 10000, // BCrypt hash size is 60 bytes | ||
| BufferItems: 64, |
There was a problem hiding this comment.
The cost in ristretto is set is kinda weird. For example, it is probably required to ignore the internal cost and use a custom cost function:
There was a problem hiding this comment.
Alright. I've added ignoreInternalCost=true but I'm not sure if custom cost function is needed at this point.
harnash
force-pushed
the
avoid_excessive_memory_allocations
branch
from
c55e16d
to
a012ef2
Compare
Codecov Report
@@ Coverage Diff @@
## master #2389 +/- ##
==========================================
- Coverage 76.60% 76.56% -0.04%
==========================================
Files 315 315
Lines 17328 17326 -2
==========================================
- Hits 13274 13266 -8
- Misses 3120 3124 +4
- Partials 934 936 +2
Continue to review full report at Codecov.
|
aeneasr
reviewed
Apr 14, 2022
peturgeorgievv
pushed a commit
to senteca/kratos-fork
that referenced
this pull request
Jun 30, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes internal storage for HIBP API call results to use ristretto cache instead of plain map to avoid excessive memory allocations when handling substantial load (each checked hash is stored indefinitely in memory).
Related issue(s)
Checklist
introduces a new feature.
contributing code guidelines.
vulnerability. If this pull request addresses a security. vulnerability, I
confirm that I got green light (please contact
[email protected]) from the maintainers to push
the changes.
works.
Further Comments
This patch was running for a few days on our cluster and it looks like memory usage is not increasing and stabilised around 70MB which is great :-)