Switch to a binary format for internal record keys

[rodesai]

When processing a query with a GROUP BY clause containing multiple columns, KSQL generates a new key for each record so that streams can repartition and aggregate according to the GROUP BY clause. The new key is the values in the GROUP BY columns concatenated together separated by the string "|+|". If the values themselves contain this separator then the resulting key may be ambiguous.

For example, consider the following query:

CREATE TABLE AS SELECT A, B, count(*) FROM STREAM FOO GROUP BY A, B

And the following two records:

{..."A":"foo|+|bar", "B":"baz"...}
{..."A":"foo", "B":"bar|+|baz"...}

Both records will take the key "foo|+|bar|+|baz", and the result of the aggregate will be incorrect.

[rodesai]

Fixing this will require using some sort of binary format for internal record key. Some options:
- fields with a size prefix
- avro

This may also require that we track key schemas per topic. It may not be reasonable to expect source topics to bey keyed according to this internal format (We currently have that expectation but feel thats ok because its reasonable to expect a string key). This means we would need to support both some external format and our internal format.

[miguno]

Related ticket: #824

[rmoff]

@rodesai is this superseded by #824? Or remains a separate issue to address?

[apurvam]

This is superseded by #824 . also see #3533 . Closing this out.