fix performance issue in scenario #2966 (part 1) #2969
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Re-employing a compression state across multiple successive compression jobs
shall minimize the amount of allocation and initialization required.
This mostly matters in situations where initialization is an overwhelming task compared to compression itself.
This can happen when the amount to compress is small,
while the compression state was expecting it to be much larger,
aka, streaming mode without the benefit of a
srcSize
hindsight.In which case,
zstd
can't rely on its automatic resource resizing capability, and initialization cost can only be made acceptable by amortizing it over a series of compressions.This diff restores a lean-initialization optimization trick that used to be present up to
v1.4.9
and was lost in commit 980f3bb .The following measurement is taken on a core
i7-9700K
(turbo disabled) withfullbench -b41
, usinggeldings.txt
as sample (a small text file). The test corresponds to a scenario usingZSTD_compressStream()
without the benefit of knowing the small sample size beforehand.Note how important is this lean-initialization optimization for small data as the compression level increases, hence the amount of resources used increases too.
Note that this PR does not completely fix #2966,
since another heavy initialization, specific to
rowHash
mode,is also present (and wasn't in
v1.4.9
).This second issue will be fixed in a separate commit.
For information :
dfast
rowHash
rowHash
rowHash
btlazy2