Align fs.ReadStream buffer pool writes to 8-byte boundary #24838

trxcllnt · 2018-12-04T22:07:29Z

fs: align fs.ReadStream buffer pool writes to 8-byte boundary

Prevents alignment issues when people create a typed array from a chunk buffer, similar to 285d8c6.

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

bnoordhuis

LGTM. It might be infinitesimally nicer to move it into a function so you can simply write pool.used = align(pool.used), with align() being:

function align(n) {
  return (n + 7) & ~7;  // Align to 8 byte boundary.
}

trxcllnt · 2018-12-04T23:15:25Z

👍 thanks @bnoordhuis, I moved the logic into a roundUpToMultipleOf8 function and updated the PR

bnoordhuis · 2018-12-05T11:12:11Z

lib/internal/fs/streams.js

+        thisPool.used = roundUpToMultipleOf8(alignedOffset);
+      } else if (toRead - bytesRead > kMinPoolSpace) {
+        alignedOffset = roundUpToMultipleOf8(start + bytesRead);
+        poolFragments.push(thisPool.slice(alignedOffset, start + toRead));


If you're only using alignedOffset inside the if/else arms, can you make it const and scope it to inside the arms?

Is start + toRead correct when alignedOffset may not be start + bytesRead? What I mean is, shouldn't start + toRead be rounded up too?

Rounding up is unsafe here, since toRead is clamped within the size of the pool on initialization. I had the same thought initially, but rounding it up caused unrelated tests to fail.

But you agree it's slicing less than before? Note that slice() takes start and end arguments (inclusive), not start and length.

After thinking about this some more, I think that the rules/constraints should be:

thisPool.used is always a multiple of 8 (the pool itself is already suitably aligned), and

this particular slice() call should only slice multiples of 8 at offsets that are multiples of 8.

(1) implies (2) because (2) caches slices for reuse by (1).

(2) can be enforced by rounding start + toRead down.

Potential pitfall: rounding up or down can result in slices smaller than kMinPoolSpace. It should check that the slice's length >= kMinPoolSpace before pushing it into poolFragments.

Yes it is slicing less than before. By rounding up the start position, we advance 8 - n_remainder bytes past where we would have sliced. In this PR, pool.used (and thisPool.used) should always be a multiple of 8, which I believe addresses point (1).

With regards to (2), the byteLength of each slice as it is now isn't guaranteed to be a multiple of 8, though I think that's also a good idea. Performance-wise, it's best to align the toRead value to the processor's cache line size, but I left it as-is to keep the surface area of this PR to a minimum. I would be happy to add this though.

Good catch on the adjusted size causing the pool fragment size to dip below kMinPoolSpace, I'll update the PR with a fix here in a bit.

bnoordhuis

LGTM % nits. CI: https://ci.nodejs.org/job/node-test-pull-request/19282/

bnoordhuis · 2018-12-06T19:16:53Z

lib/internal/fs/streams.js

-      else if (toRead - bytesRead > kMinPoolSpace)
-        poolFragments.push(thisPool.slice(start + bytesRead, start + toRead));
+      if (start + toRead === thisPool.used && thisPool === pool) {
+        const alignedOffset = thisPool.used + bytesRead - toRead;


Tiniest of nits: alignedOffset is a misnomer since it's not actually aligned (yet.)

bnoordhuis · 2018-12-06T19:19:06Z

lib/internal/fs/streams.js

+        const alignedOffset = thisPool.used + bytesRead - toRead;
+        thisPool.used = roundUpToMultipleOf8(alignedOffset);
+      } else {
+        const alignedEnd = (start + toRead) & ~7; // round down to multiple of 8


One more micro-nit: comments should be capitalized and punctuated (and normally have two spaces before the // - I expect the linter will complain.)

bnoordhuis

https://ci.nodejs.org/job/node-test-pull-request/19338/

trxcllnt · 2018-12-14T20:39:09Z

@bnoordhuis what else needs to be done here? Are those three failing checks related to this change? I tried parsing through the console output, but can't see any indication of why it's failing.

bnoordhuis · 2019-01-08T11:11:55Z

Let's run the CI again and see what happens: https://ci.nodejs.org/job/node-test-pull-request/19989/

addaleax · 2019-01-21T21:56:03Z

@trxcllnt Do you think you could rebase this? Sorry for the merge conflicts!

trxcllnt · 2019-01-21T22:21:53Z

@addaleax just resolved the conflict via the github UI, let me know if I should do anything further. Thanks!

addaleax · 2019-01-21T22:24:30Z

@trxcllnt One problem is that merge conflicts break our CI and make landing changes harder – can you use git rebase rather than a merge commit?

(We can probably figure this out another way if not, but this would definitely make things easier)

trxcllnt · 2019-01-21T23:50:17Z

@addaleax sure thing -- rebased from upstream/master and resolved the conflict locally. I'm always wary of resolving conflicts during a rebase, so let me know if it doesn't look right.

trxcllnt · 2019-03-31T02:55:48Z

@bnoordhuis @addaleax anything left to do here? The node folks in the Apache Arrow project would appreciate if we could merge this soon 🙏 since we have a workaround in place that has to resort to copying chunks if the offset isn't aligned. It can be a nasty/hidden performance cliff for users if they're doing a lot of file I/O. Let me know if there's anything else I can do ❤️

addaleax · 2019-03-31T09:18:16Z

@trxcllnt It looks like the rebase went wrong somehow? There’s 22000 added lines in this PR now…

Generally, there’s nothing in the way of landing it, and bumping the thread is the exact right thing to do :)

trxcllnt · 2019-04-01T00:28:23Z

@addaleax thanks for the heads up! should be fixed now :-)

nodejs-github-bot · 2019-04-01T07:09:15Z

CI: https://ci.nodejs.org/job/node-test-pull-request/22092/

Prevents alignment issues when creating a typed array from a buffer. Fixes: nodejs#24817

trxcllnt · 2019-05-01T00:52:24Z

@addaleax bump :-]

nodejs-github-bot · 2019-05-01T22:04:24Z

CI: https://ci.nodejs.org/job/node-test-pull-request/22866/

addaleax · 2019-05-02T00:09:53Z

Landed in b884ceb 🎉 Thanks for the PR!

Prevents alignment issues when creating a typed array from a buffer. Fixes: #24817 PR-URL: #24838 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

nodejs-github-bot added the fs Issues and PRs related to the fs subsystem / file system. label Dec 4, 2018

bnoordhuis approved these changes Dec 4, 2018

View reviewed changes

trxcllnt force-pushed the bugfix/24817 branch from e44cc01 to 7aaae68 Compare December 4, 2018 23:13

trxcllnt force-pushed the bugfix/24817 branch 2 times, most recently from 3c5ec58 to f530dc5 Compare December 5, 2018 05:30

bnoordhuis reviewed Dec 5, 2018

View reviewed changes

trxcllnt force-pushed the bugfix/24817 branch 2 times, most recently from e13a20c to 0d29ef3 Compare December 5, 2018 23:40

bnoordhuis approved these changes Dec 6, 2018

View reviewed changes

trxcllnt force-pushed the bugfix/24817 branch 2 times, most recently from 899e641 to 73d1ca2 Compare December 6, 2018 23:34

bnoordhuis approved these changes Dec 8, 2018

View reviewed changes

jasnell approved these changes Jan 8, 2019

View reviewed changes

trxcllnt force-pushed the bugfix/24817 branch from 01b5fd8 to 82ec8d3 Compare January 21, 2019 23:47

trxcllnt force-pushed the bugfix/24817 branch 2 times, most recently from d6d9c3f to bdc2ba4 Compare April 1, 2019 00:26

fs: align fs.ReadStream buffer pool writes to 8-byte boundary

f2e1ed3

Prevents alignment issues when creating a typed array from a buffer. Fixes: nodejs#24817

trxcllnt force-pushed the bugfix/24817 branch from bdc2ba4 to f2e1ed3 Compare April 14, 2019 00:24

addaleax approved these changes May 1, 2019

View reviewed changes

addaleax closed this May 2, 2019

targos mentioned this pull request May 6, 2019

v12.2.0 proposal #27578

Merged

Zclhlmgqzc mentioned this pull request Oct 2, 2020

[Bug report] fs.createReadStream set highWaterMark, the same file get different result #35465

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Align fs.ReadStream buffer pool writes to 8-byte boundary #24838

Align fs.ReadStream buffer pool writes to 8-byte boundary #24838

trxcllnt commented Dec 4, 2018

bnoordhuis left a comment

trxcllnt commented Dec 4, 2018

bnoordhuis Dec 5, 2018

trxcllnt Dec 5, 2018

bnoordhuis Dec 5, 2018

trxcllnt Dec 5, 2018 •

edited

Loading

bnoordhuis left a comment

bnoordhuis Dec 6, 2018

bnoordhuis Dec 6, 2018

bnoordhuis left a comment

trxcllnt commented Dec 14, 2018

bnoordhuis commented Jan 8, 2019

addaleax commented Jan 21, 2019

trxcllnt commented Jan 21, 2019

addaleax commented Jan 21, 2019

trxcllnt commented Jan 21, 2019

trxcllnt commented Mar 31, 2019 •

edited

Loading

addaleax commented Mar 31, 2019

trxcllnt commented Apr 1, 2019

nodejs-github-bot commented Apr 1, 2019

trxcllnt commented May 1, 2019

nodejs-github-bot commented May 1, 2019

addaleax commented May 2, 2019

Align fs.ReadStream buffer pool writes to 8-byte boundary #24838

Align fs.ReadStream buffer pool writes to 8-byte boundary #24838

Conversation

trxcllnt commented Dec 4, 2018

Checklist

bnoordhuis left a comment

Choose a reason for hiding this comment

trxcllnt commented Dec 4, 2018

bnoordhuis Dec 5, 2018

Choose a reason for hiding this comment

trxcllnt Dec 5, 2018

Choose a reason for hiding this comment

bnoordhuis Dec 5, 2018

Choose a reason for hiding this comment

trxcllnt Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

bnoordhuis left a comment

Choose a reason for hiding this comment

bnoordhuis Dec 6, 2018

Choose a reason for hiding this comment

bnoordhuis Dec 6, 2018

Choose a reason for hiding this comment

bnoordhuis left a comment

Choose a reason for hiding this comment

trxcllnt commented Dec 14, 2018

bnoordhuis commented Jan 8, 2019

addaleax commented Jan 21, 2019

trxcllnt commented Jan 21, 2019

addaleax commented Jan 21, 2019

trxcllnt commented Jan 21, 2019

trxcllnt commented Mar 31, 2019 • edited Loading

addaleax commented Mar 31, 2019

trxcllnt commented Apr 1, 2019

nodejs-github-bot commented Apr 1, 2019

trxcllnt commented May 1, 2019

nodejs-github-bot commented May 1, 2019

addaleax commented May 2, 2019

trxcllnt Dec 5, 2018 •

edited

Loading

trxcllnt commented Mar 31, 2019 •

edited

Loading