Upload Large file

[pyxiscloud]

Hello, I'm trying upload big file (40Mb) and get this error:
buffer.js:11
super(arg1, arg2, arg3);
^

RangeError: Array buffer allocation failed
at Buffer.Uint8Array (native)
at FastBuffer (buffer.js:11:5)
at createUnsafeBuffer (buffer.js:38:12)
at allocate (buffer.js:181:12)
at Function.Buffer.allocUnsafe (buffer.js:141:10)
at Function.Buffer.concat (buffer.js:309:23)
at FileStream. (/opt/pyxis_server/node_modules/express-fileupload/lib/index.js:27:24)
at emitOne (events.js:96:13)
at FileStream.emit (events.js:188:7)
at readableAddChunk (_stream_readable.js:176:18)

Uploading 10Mb file is ok.

[richardgirges]

@reejinbouk can you show me how you are setting up express-fileupload with express?

[richardgirges]

PR #22 should have resolved this.

[sant123]

Hi, I'm trying to upload a file of 2GB, but according with this nodejs/node#6560 2GB is the limit size. What can I do then? Splitting the file in chunks?

Thanks.

[mikeatlas]

@richardgirges PR #22 did indeed improve the performance during the transfer, but the same error occurs for me (uploading 4gb) at the end of the transfer:

buffer.js:11
    super(arg1, arg2, arg3);
    ^

RangeError: Invalid typed array length
    at Buffer.Uint8Array (native)
    at FastBuffer (buffer.js:11:5)
    at createUnsafeBuffer (buffer.js:38:12)
    at allocate (buffer.js:181:12)
    at Function.Buffer.allocUnsafe (buffer.js:141:10)
    at Function.Buffer.concat (buffer.js:309:23)
    at FileStream.<anonymous> (node_modules/express-fileupload/lib/index.js:84:26)
    at emitNone (events.js:91:20)
    at FileStream.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

I'm running the latest express-fileupload, version 0.1.3.

[mikeatlas]

This points to Buffer.concat as the culprit... the data type underneath isn't ever going to be big enough.

Perhaps, in the case of large files instead of on('data') calls collected into buffers which are finally concatenated into a final buffer in on('end'), an alternative would be to write the bytes to a temporary file on disk and then point to the final file, rather than keep all the bytes in memory for concatenation at the end of transfer. Perhaps I'll do a proof of concept PR.