-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce stack usage of block splitter #2780
Conversation
99ddad5
to
e63fa39
Compare
lib/compress/zstd_compress.c
Outdated
/* block splitter ctx */ | ||
if (ZSTD_CParams_useBlockSplitter(¶ms->cParams)) { | ||
/* Silence -Wcast-align with cast to void* */ | ||
zc->blockSplitCtx = (ZSTD_blockSplitCtx*)(void*)ZSTD_cwksp_reserve_buffer(ws, sizeof(ZSTD_blockSplitCtx)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-Wcast-align
has a point here. You'll need correct alignment here or certain platforms will fail.
This probably needs to go in the ZSTD_cwksp_reserve_object()
space, or if that doesn't work just put it directly into the CCtx.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah you're right. I think this will just rest directly in the CCtx. If I want to save space when we're not using block splitting, a wksp object doesn't seem to work properly with the different cctx reuse cases, since objects are assumed to remain the same throughout compressions.
If I wanted to make it work somehow, it would have to be the same as the other objects in that they stay the same throughout different cctx re-uses. And if we have to do that anyways, we may as well just keep it in the cctx. Though, this does mean we will just have to eat the >1K additional heap cost.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Though, this does mean we will just have to eat the >1K additional heap cost.
That should be fine, thats much less than the total CCtx allocation. E.g. we have 500KB space for sequences, which is way overkill in most cases. So there is lower hanging fruit.
e63fa39
to
7f0f526
Compare
Rebased on top of #2788. For review, just consider the last commit in the stack. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me!
Before merging, you should rebase on top of dev
, so you don't get any extra commits in the diff.
7f0f526
to
1d8143c
Compare
This PR moves some of the block splitter stuff that was using too much stack space
I see no speed delta for levels 16-19 (have block splitter enabled by default) on
enwik7
withgcc
.Todo: Fix fuzzer errors