Fix Determinism Bug: Avoid Reducing Indices to Reserved Values #2850
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, if an index was equal to
reducerValue + 1
, it would get remappedduring index reduction to 1 i.e.
ZSTD_DUBT_UNSORTED_MARK
. This can affectthe parsing of the input slightly, by causing tree nodes to be nullified when
they otherwise wouldn't be. This hardly matters from a correctness or
efficiency perspective, but it does impact determinism.
So this Pull Request changes index reduction to avoid mapping indices to
collide with
ZSTD_DUBT_UNSORTED_MARK
.I am somewhat concerned that
ZSTD_reduceTable_internal()
will be slowernow. I'm not sure how important the speed of this is in practive, but it
looks like it's written to be auto-vectorized, and I'm not sure the new
version is similarly vectorizable.
Credit to OSS-Fuzz for discovery.