RangeError: Invalid typed array length, for read stream on 62 Gb file

[Orionsg]

• Version: v4.4.3
• Platform: Windows 32 bit / Windows 64 bit
• Subsystem:

I have made a script to parse a 62 Gb datafile. It creates a readstream utf8 and and uses 'data' event to parse the blocks of data read. The script works without any trouble for smaller files. However, when I try to execute it on the full 62 Gb datafile, the script terminates right away with this output:

"buffer.js:23
const ui8 = new Uint8Array(size);
^

RangeError: Invalid typed array length
at new Uint8Array (native)
at createBuffer (buffer.js:23:15)
at allocate (buffer.js:98:12)
at new Buffer (buffer.js:53:12)
at Object.fs.readFileSync (fs.js:419:16)
at Object.Module._extensions..js (module.js:415:20)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Function.Module.runMain (module.js:441:10)
at startup (node.js:139:18)"

The error occurs when executing the script under both Windows 7 32 bit and Windows 10 64 bit.

[JacksonTian]

The Buffer can't bigger than 2147483647 bytes. please use stream to process it.

[Orionsg]

As mentioned I do use a readstream:
var rs = fs.createReadStream(infile);

Perhaps there is some variable in the buffer module that even for streaming gets set according to the total length of the file to be parsed. This variable must be an integer, but should be bigint in this day and age with 64 bit for larger memory handling and file handling.

[bnoordhuis]

@Orionsg The stack trace suggests you're using fs.readFileSync():

at Object.fs.readFileSync (fs.js:419:16)

That line number indeed corresponds with a call to new Buffer(size) in v4.4.3.

[Orionsg]

I do also use the fs.existsSync function to check for the existence of some files, e.g. the existence of the file to be parsed. I now tried to comment out fs.existsSync on the infile that is the large one. That did not make any difference.

I searched my source file and do not have any readFileSync statements in it. Could it be that fs.existsSync indirectly call readFileSync when checking for the existence of a file, and perhaps even give and error if any other file in the folder is large (unlikely, as that sounds).

If you think that the fs.existsSync is the culprit, even when used for other, smaller files, I can change the code to not use that function at all. I just use it for checking for and handling of log and abort files that are very small.

I do believe there could a problem with the streaming function itself for very large files.

I am quite new to node.js, but I will see if I can find a way to debug through the code to see where the offending statement in my script is.

[bnoordhuis]

How exactly are you invoking node.js? The stack trace looks like you're trying to run or require() a really big file.

Perhaps you can post a reduced version of your script and describe the steps for us to reproduce?

[Orionsg]

I withdraw this issue, much embarrassed.
Thanks for leading me to the obvious.

I am developing the script in Visual Studio and have been executing it directly there on smaller test files numerous times. New to node.js that's been nearly all my interaction with node.js so far. When wanting to test on production file, I moved the application over and ran it from the command prompt. However, inexperienced and very eager to run it, I filled in "node" and my parameters for various files, forgetting the js file itself -- how very silly an oversight. The result was that node.js read the infile as the script to be executed which of course did not work, and for each subsequent trial I just used the command buffer to recall the last executed, erroneous command again and again.

I apologize for wasting your time and, again, thanks for pointing me to the obvious.

[bnoordhuis]

No problem. Glad to hear it's working for you now.