Utf8Stream

(Since 1.7.0) This utility is a Transform text-to-text stream. It reads text as chunks of Buffer and transforms it into utf8 text respecting multibyte boundaries. In the case of chunks of string, it serves as a pass-through.

This class can be used as a base class for Transform streams dealing with text as input. Parser and jsonl/Parser are based on it. Verifier borrows the technique.

Internally this class uses String Decoder to do its magic.

Introduction

The simple example (streaming from a file):

const Utf8Stream = require('stream-json/utils/Utf8Stream');
const utf8Stream = new Utf8Stream();

const fs = require('fs');

const pipeline = fs.createReadStream('european-languages.json').pipe(utf8Stream);
// clients of this pipeline will not see breaks inside multibyte characters

API

The module returns the constructor of Utf8Stream. Being a Transform stream it doesn't have any special interfaces. The only thing required is to configure it during the construction.

constructor([options])

options is an optional object described in details in node.js' Stream documentation. The writing part of the stream always works in text mode.

The constructor defines the following instance variables:

• this._buffer is a string variable, which keeps the current input. The initial value: ''.

_processBuffer(callback)

This method is called from _transform() when new data has arrived. It can inspect this._buffer and should clear it after it was processed. By default, this._buffer value is pushed downstream.

callback is the same argument that was passed in _transform() with the same semantics. It should be called according to the documentation: with an error value in case of some error, and with null otherwise.

Users can override this method to produce different effects.

_flushInput()

This method can populate this._buffer with the rest of input data. It is meant to be called from _flush().

_flush(callback)

This is a documented method of the Transform implementation: _flush(). In this class it is defined like that:

_flush(callback) {
  this._flushInput();
  this._processBuffer(callback);
}

Users can override it for different side-effects.