Binary data transfer

[tshemsedinov]

• Support streaming but with chunks, because connection should not be blocked by single large data transmission stream
• Client and server will arrange chunk size in handshake packets
• Stream initialization packet will contain metadata about data type (file, streaming video or audio, etc.), file name (optional), total size (optional)
• Both sides will be able to suspend, resume and terminate stream

[aqrln]

Depends on #34

[aqrln]

Binary streaming in JSTP will be built on top of Node.js Stream API. We need to subclass stream.Writable (which will be created when a side initiates binary streaming) and stream.Readable (which will be created on another side) and implement a handful of methods, Node.js will do the rest of magic (including suspend/resume functionality) for us.

[aqrln]

Client and server will arrange chunk size in handshake packets

It's better to arrange it in stream initialization packets. The default value should be based on the highWaterMark value of the underlying socket stream.

[aqrln]

Stream initialization packet:

{ stream: [42, chunkSize?], metadata: [totalSize?, streamType?, fileName?] }

Stream initialization acknowledgement packet:

{ callback: [42], ok: ['streamId'] }

or

{ callback: [42], error: [ /* error array */ ] }

Chunk metadata packet (immediately followed by pure binary data):

{ data: [234, 'streamId'] }

Last chunk metadata packet:

{ end: [234, 'streamId', chunkSize] }

Additional packets for stream state change notifications will be specified later.

[aqrln]

Some notes on remote methods accepting streams as parameters.

On low level, each method accepting a stream will actually accept a string — stream identifier (streamId in the above comment). So basically one would need to perform the following steps:

1. Create a binary stream (stream packet) and obtain streamId.
2. Call a remote method and pass streamId.
3. Start streaming data.

The remote method will be able to obtain a stream by streamId:

...
const stream = connection.streams[streamId];
if (!stream) return callback(new Error('unknown stream'));

stream.on('data', () => { ... });
stream.on('end', () => { ... });
...

After #19 and metarhia/impress#608 it will become much nicer to use. The signature of a remote method will specify that a parameter is a stream, and a client will know about that too after retrieving introspection. Thus to call such a method one would just call a method of a proxy object passing any arbitrary readable stream:

gallery.uploadPhoto(fs.createReadStream('./photo.jpg'));

The proxy object will perform the above steps behind the curtain and pipe the passed stream into an internal JSTP binary stream.

Accordingly, a method won't need to obtain the stream explicitly, it will already be passed one instead of streamId.