P2P shuffle: pickle tiny buffers into monolithic bytes objects #8321
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In dataframe shuffle with shards >= 128 kiB each, avoid one deep copy of the buffers by letting the network stack transfer the shards as individual buffers
In array shuffle with shards < 128 kiB each, avoid the overhead of sending the buffers separately through the network stack by pickling everything into a single opaque bytes object ahead of time
Blocked by and incorporates Zero-copy array shuffle #8282
See only the last commit for review.
Fixes performance regression with small chunks introduced by Zero-copy array shuffle #8282
See performance measures in Speed up network transfer for small buffers #8318