-
Notifications
You must be signed in to change notification settings - Fork 6
Streaming Data Message
Previous versions of the framework operated on messages of type string which - in most cases - contained an application specific JSON format. As the number of use cases increased and not everything travelling through SPQR pipelines was a perfect match for a JSON representation the message type was changed to an array of bytes. A nice side-effect is a major increase in processing speed as the travelling content is represented by number only.
As it may be sufficient to have some meta data on each message, like ingestion time or creation time or any other attribute we actually cannot think of, the byte array was wrapped into a specific message type that can easily be extended: StreamingDataMessage.
Every implementation of pipeline component must take care of the message content and convert it into a processable representation.
SPQR - stream processing and querying in realtime by Otto Group