feat: align buffers in 2.1 files #3101
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3101 +/- ##
==========================================
- Coverage 77.16% 77.14% -0.02%
==========================================
Files 240 240
Lines 79764 79833 +69
Branches 79764 79833 +69
==========================================
+ Hits 61548 61586 +38
- Misses 15071 15082 +11
- Partials 3145 3165 +20
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| @@ -376,17 +377,26 @@ impl DecodePageTask for DecodeMiniBlockTask { | |||
| for chunk in self.chunks.into_iter() { | |||
| // We always decode the entire chunk | |||
| let buf = chunk.data.into_buffer(); | |||
| // The first 4 bytes describe the size of the rep/def buffers | |||
| // The first 6 bytes describe the size of the remaining buffers | |||
There was a problem hiding this comment.
judging from the compress code, looks like the size of bytes_rep and bytes_def are at the end of the chunk now
There was a problem hiding this comment.
oh, sorry, only the comments in the compress code states that the size of the rep buffer and def buffer are put at the end.
// Each chunk ends with the size of the rep buffer (2 bytes) and the size of
// the def buffer (2 bytes). These are put at the end so as to not disturb
// the alignment guarantees given to the compressor.
There was a problem hiding this comment.
Oops, I'll clean that up. At one point in this PR they were at the end but then I put them back at the start and didn't repair this comment.
Co-authored-by: broccoliSpicy <[email protected]>
…he buffer alignment into account
This ensures that all file buffers are aligned to 64 bytes (these are generally 8MiB buffers and so 64 bytes is trivial)
We also ensure that all mini-block chunks are aligned to 8 bytes. This introduced a bit more padding but makes it easier to do things like store offsets into mini-block chunks.