Skip to content
This repository has been archived by the owner on Apr 22, 2023. It is now read-only.

fs: pool size coincide with ReadStream bufferSize #4518

Closed
wants to merge 2 commits into from

Conversation

shigeki
Copy link

@shigeki shigeki commented Jan 4, 2013

pool size of file reading in ReadableStream was statically set 40k. It can be adjustable with a bufferSize option for performace tuning of file read.
Benchmarking results in reading/writing file stream of 100Mbyte are below.

//  fs_stream_bench.js
var fs = require('fs');
var infile = '/home/ohtsu/tmp/100M.file';
var outfile = infile + '.out';
var kB = 1024;
var bufferSizeList = [2 * 64 * kB, 3 * 64 * kB, 4 * 64 * kB, 5 * 64 * kB];

function fsStreamBench(bufferSize) {
  var option = {};
  if (bufferSize) option.bufferSize = bufferSize;
  if (fs.existsSync(outfile)) fs.unlinkSync(outfile);

  var rstream = fs.createReadStream(infile, option);
  var wstream = fs.createWriteStream(outfile);

  var start = Date.now();
  rstream.pipe(wstream);

  wstream.on('finish', function() {
    var bufSize = rstream._readableState.bufferSize;
    console.log('bufferSize:', bufSize, ', time:', Date.now() - start);
    if (bufferSizeList.length) fsStreamBench(bufferSizeList.shift());
  });
}

fsStreamBench();
# node-v0.9.5
> ./node ~/tmp/fs_stream_bench.js
bufferSize: 65536 , time: 307
bufferSize: 131072 , time: 287
bufferSize: 196608 , time: 274
bufferSize: 262144 , time: 277
bufferSize: 327680 , time: 320

# after this commit
> ./node ~/tmp/fs_stream_bench.js
bufferSize: 65536 , time: 255
bufferSize: 131072 , time: 196
bufferSize: 196608 , time: 156
bufferSize: 262144 , time: 170
bufferSize: 327680 , time: 139

And another commit to remove unsed bufferSize option in WriteStream was added.

@@ -1370,7 +1369,7 @@ fs.realpath = function realpath(p, cache, cb) {

var pool;

function allocNewPool() {
function allocNewPool(kPoolSize) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name it poolSize, kSomething is for constants.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I will change it.

Shigeki Ohtsu added 2 commits January 5, 2013 02:26
pool size of file reading in ReadableStream can be adjustable
with a bufferSize option.
No bufferSize option in stream.Writable and fs.WriteStream
@shigeki
Copy link
Author

shigeki commented Jan 4, 2013

@bnoordhuis Thanks for your reviewing. The variable name was changed and the commit was forced to be pushed.

@bnoordhuis
Copy link
Member

Thanks Shigeki, landed in 829f96b and 2b6c561.

@bnoordhuis bnoordhuis closed this Jan 4, 2013
gibfahn pushed a commit to ibmruntimes/node that referenced this pull request Apr 26, 2016
The following significant (semver-major) changes have been made since the
previous Node v5.0.0 release.

* Buffer
  * New Buffer constructors have been added
    [nodejs#4682](nodejs/node#4682)
  * Previously deprecated Buffer APIs are removed
    [nodejs#5048](nodejs/node#5048),
    [nodejs#4594](nodejs/node#4594)
  * Improved error handling [nodejs#4514](nodejs/node#4514)
* Cluster
  * Worker emitted as first argument in 'message' event
    [nodejs#5361](nodejs/node#5361).
* Crypto
  * Improved error handling [nodejs#3100](nodejs/node#3100),
    [nodejs#5611](nodejs/node#5611)
  * Simplified Certificate class bindings
    [nodejs#5382](nodejs/node#5382)
  * Improved control over FIPS mode
    [nodejs#5181](nodejs/node#5181)
  * pbkdf2 digest overloading is deprecated
    [nodejs#4047](nodejs/node#4047)
* Dependencies
  * Reintroduce shared c-ares build support
    [nodejs#5775](nodejs/node#5775).
  * V8 updated to 5.0.71.31 [nodejs#6111](nodejs/node#6111).
* DNS
  * Add resolvePtr API to query plain DNS PTR records
    [nodejs#4921](nodejs/node#4921).
* Domains
  * Clear stack when no error handler
  [nodejs#4659](nodejs/node#4659).
* File System
  * The `fs.realpath()` and `fs.realpathSync()` methods have been updated
    to use a more efficient libuv implementation. This change includes the
    removal of the `cache` argument and the method can throw new errors
    [nodejs#3594](nodejs/node#3594)
  * FS apis can now accept and return paths as Buffers
    [nodejs#5616](nodejs/node#5616).
  * Error handling and type checking improvements
    [nodejs#5616](nodejs/node#5616),
    [nodejs#5590](nodejs/node#5590),
    [nodejs#4518](nodejs/node#4518),
    [nodejs#3917](nodejs/node#3917).
  * fs.read's string interface is deprecated
    [nodejs#4525](nodejs/node#4525)
* HTTP
  * 'clientError' can now be used to return custom errors from an
    HTTP server [nodejs#4557](nodejs/node#4557).
* Modules
  * Current directory is now prioritized for local lookups
    [nodejs#5689](nodejs/node#5689)
  * Symbolic links are preserved when requiring modules
    [nodejs#5950](nodejs/node#5950)
* Net
  * DNS hints no longer implicitly set
    [nodejs#6021](nodejs/node#6021).
  * Improved error handling and type checking
    [nodejs#5981](nodejs/node#5981),
    [nodejs#5733](nodejs/node#5733),
    [nodejs#2904](nodejs/node#2904)
* Path
  * Improved type checking [nodejs#5348](nodejs/node#5348).
* Process
  * Introduce process warnings API
    [nodejs#4782](nodejs/node#4782).
  * Throw exception when non-function passed to nextTick
    [nodejs#3860](nodejs/node#3860).
* Readline
  * Emit key info unconditionally
    [nodejs#6024](nodejs/node#6024)
* REPL
  * Assignment to `_` will emit a warning.
    [nodejs#5535](nodejs/node#5535)
* Timers
  * Fail early when callback is not a function
    [nodejs#4362](nodejs/node#4362)
* TLS
  * Rename 'clientError' to 'tlsClientError'
    [nodejs#4557](nodejs/node#4557)
  * SHA1 used for sessionIdContext
    [nodejs#3866](nodejs/node#3866)
* TTY
  * Previously deprecated setRawMode wrapper is removed
    [nodejs#2528](nodejs/node#2528).
* Util
  * Changes to Error object formatting
    [nodejs#4582](nodejs/node#4582).
* Windows
  * Windows XP and Vista are no longer supported
    [nodejs#5167](nodejs/node#5167),
    [nodejs#5167](nodejs/node#5167).
gibfahn pushed a commit to ibmruntimes/node that referenced this pull request Apr 27, 2016
The following significant (semver-major) changes have been made since the
previous Node v5.0.0 release.

* Buffer
  * New Buffer constructors have been added
    [nodejs#4682](nodejs/node#4682)
  * Previously deprecated Buffer APIs are removed
    [nodejs#5048](nodejs/node#5048),
    [nodejs#4594](nodejs/node#4594)
  * Improved error handling [nodejs#4514](nodejs/node#4514)
* Cluster
  * Worker emitted as first argument in 'message' event
    [nodejs#5361](nodejs/node#5361).
* Crypto
  * Improved error handling [nodejs#3100](nodejs/node#3100),
    [nodejs#5611](nodejs/node#5611)
  * Simplified Certificate class bindings
    [nodejs#5382](nodejs/node#5382)
  * Improved control over FIPS mode
    [nodejs#5181](nodejs/node#5181)
  * pbkdf2 digest overloading is deprecated
    [nodejs#4047](nodejs/node#4047)
* Dependencies
  * Reintroduce shared c-ares build support
    [nodejs#5775](nodejs/node#5775).
  * V8 updated to 5.0.71.31 [nodejs#6111](nodejs/node#6111).
* DNS
  * Add resolvePtr API to query plain DNS PTR records
    [nodejs#4921](nodejs/node#4921).
* Domains
  * Clear stack when no error handler
  [nodejs#4659](nodejs/node#4659).
* File System
  * The `fs.realpath()` and `fs.realpathSync()` methods have been updated
    to use a more efficient libuv implementation. This change includes the
    removal of the `cache` argument and the method can throw new errors
    [nodejs#3594](nodejs/node#3594)
  * FS apis can now accept and return paths as Buffers
    [nodejs#5616](nodejs/node#5616).
  * Error handling and type checking improvements
    [nodejs#5616](nodejs/node#5616),
    [nodejs#5590](nodejs/node#5590),
    [nodejs#4518](nodejs/node#4518),
    [nodejs#3917](nodejs/node#3917).
  * fs.read's string interface is deprecated
    [nodejs#4525](nodejs/node#4525)
* HTTP
  * 'clientError' can now be used to return custom errors from an
    HTTP server [nodejs#4557](nodejs/node#4557).
* Modules
  * Current directory is now prioritized for local lookups
    [nodejs#5689](nodejs/node#5689)
  * Symbolic links are preserved when requiring modules
    [nodejs#5950](nodejs/node#5950)
* Net
  * DNS hints no longer implicitly set
    [nodejs#6021](nodejs/node#6021).
  * Improved error handling and type checking
    [nodejs#5981](nodejs/node#5981),
    [nodejs#5733](nodejs/node#5733),
    [nodejs#2904](nodejs/node#2904)
* OS X
  * MACOSX_DEPLOYMENT_TARGET has been bumped up to 10.7
    [nodejs#6402](nodejs/node#6402).
* Path
  * Improved type checking [nodejs#5348](nodejs/node#5348).
* Process
  * Introduce process warnings API
    [nodejs#4782](nodejs/node#4782).
  * Throw exception when non-function passed to nextTick
    [nodejs#3860](nodejs/node#3860).
* Readline
  * Emit key info unconditionally
    [nodejs#6024](nodejs/node#6024)
* REPL
  * Assignment to `_` will emit a warning.
    [nodejs#5535](nodejs/node#5535)
* Timers
  * Fail early when callback is not a function
    [nodejs#4362](nodejs/node#4362)
* TLS
  * Rename 'clientError' to 'tlsClientError'
    [nodejs#4557](nodejs/node#4557)
  * SHA1 used for sessionIdContext
    [nodejs#3866](nodejs/node#3866)
* TTY
  * Previously deprecated setRawMode wrapper is removed
    [nodejs#2528](nodejs/node#2528).
* Util
  * Changes to Error object formatting
    [nodejs#4582](nodejs/node#4582).
* Windows
  * Windows XP and Vista are no longer supported
    [nodejs#5167](nodejs/node#5167),
    [nodejs#5167](nodejs/node#5167).
gibfahn pushed a commit to ibmruntimes/node that referenced this pull request May 6, 2016
The following significant (semver-major) changes have been made since the
previous Node v5.0.0 release.

* Buffer
  * New Buffer constructors have been added
    [nodejs#4682](nodejs/node#4682)
  * Previously deprecated Buffer APIs are removed
    [nodejs#5048](nodejs/node#5048),
    [nodejs#4594](nodejs/node#4594)
  * Improved error handling [nodejs#4514](nodejs/node#4514)
* Cluster
  * Worker emitted as first argument in 'message' event
    [nodejs#5361](nodejs/node#5361).
* Crypto
  * Improved error handling [nodejs#3100](nodejs/node#3100),
    [nodejs#5611](nodejs/node#5611)
  * Simplified Certificate class bindings
    [nodejs#5382](nodejs/node#5382)
  * Improved control over FIPS mode
    [nodejs#5181](nodejs/node#5181)
  * pbkdf2 digest overloading is deprecated
    [nodejs#4047](nodejs/node#4047)
* Dependencies
  * Reintroduce shared c-ares build support
    [nodejs#5775](nodejs/node#5775).
  * V8 updated to 5.0.71.31 [nodejs#6111](nodejs/node#6111).
* DNS
  * Add resolvePtr API to query plain DNS PTR records
    [nodejs#4921](nodejs/node#4921).
* Domains
  * Clear stack when no error handler
  [nodejs#4659](nodejs/node#4659).
* File System
  * The `fs.realpath()` and `fs.realpathSync()` methods have been updated
    to use a more efficient libuv implementation. This change includes the
    removal of the `cache` argument and the method can throw new errors
    [nodejs#3594](nodejs/node#3594)
  * FS apis can now accept and return paths as Buffers
    [nodejs#5616](nodejs/node#5616).
  * Error handling and type checking improvements
    [nodejs#5616](nodejs/node#5616),
    [nodejs#5590](nodejs/node#5590),
    [nodejs#4518](nodejs/node#4518),
    [nodejs#3917](nodejs/node#3917).
  * fs.read's string interface is deprecated
    [nodejs#4525](nodejs/node#4525)
* HTTP
  * 'clientError' can now be used to return custom errors from an
    HTTP server [nodejs#4557](nodejs/node#4557).
* Modules
  * Current directory is now prioritized for local lookups
    [nodejs#5689](nodejs/node#5689)
  * Symbolic links are preserved when requiring modules
    [nodejs#5950](nodejs/node#5950)
* Net
  * DNS hints no longer implicitly set
    [nodejs#6021](nodejs/node#6021).
  * Improved error handling and type checking
    [nodejs#5981](nodejs/node#5981),
    [nodejs#5733](nodejs/node#5733),
    [nodejs#2904](nodejs/node#2904)
* OS X
  * MACOSX_DEPLOYMENT_TARGET has been bumped up to 10.7
    [nodejs#6402](nodejs/node#6402).
* Path
  * Improved type checking [nodejs#5348](nodejs/node#5348).
* Process
  * Introduce process warnings API
    [nodejs#4782](nodejs/node#4782).
  * Throw exception when non-function passed to nextTick
    [nodejs#3860](nodejs/node#3860).
* Readline
  * Emit key info unconditionally
    [nodejs#6024](nodejs/node#6024)
* REPL
  * Assignment to `_` will emit a warning.
    [nodejs#5535](nodejs/node#5535)
* Timers
  * Fail early when callback is not a function
    [nodejs#4362](nodejs/node#4362)
* TLS
  * Rename 'clientError' to 'tlsClientError'
    [nodejs#4557](nodejs/node#4557)
  * SHA1 used for sessionIdContext
    [nodejs#3866](nodejs/node#3866)
* TTY
  * Previously deprecated setRawMode wrapper is removed
    [nodejs#2528](nodejs/node#2528).
* Util
  * Changes to Error object formatting
    [nodejs#4582](nodejs/node#4582).
* Windows
  * Windows XP and Vista are no longer supported
    [nodejs#5167](nodejs/node#5167),
    [nodejs#5167](nodejs/node#5167).
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants