forked from confluentinc/ksql
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
refactor: remove
Struct
from Format interface
Following on from confluentinc#6200, this change changes the `Format` interface so that the created `Serde` is `Serde<List<?>>` rather than the previous `Serde<Struct>`, i.e. the format returns serde capable of returning a list of values, rather than requiring those values in a Connect `Struct`. This is part of the work of moving away from using Connect's `Struct` type internally. All the 'ConnectFormat' sub-classes still internally using `Serde<Struct>` for now, as that's what the Connect code returns. This change is not trying to fix this. `DELIMITED` and `KAFKA` formats no longer know about any `Connect` types, i.e. no `Struct` and no `Schema`. Yay. In the engine the key is still passed around as a `Struct`. This change is not trying to fix this. So there is code to convert the returned `List<?>` into the key `Struct`. However, this will go once the key is no longer a `Struct`, and its only a single, primitive field, so the overhead is very low, and the `KAFKA` format was doing this already internally anyway, (so no change in perf). Of course, any one value in the returned `List<?>` can still be a Connect `Struct` if the column type is a struct. This change is not trying to fix this. This should be fixed later. Serde performance for JSON and AVRO remains the same, (no benchmark available for other formats yet). Previous benchmark results (ran locally): ``` Benchmark (schemaName) (serializationFormat) Mode Cnt Score Error Units SerdeBenchmark.deserialize impressions JSON avgt 3 1.148 ± 0.205 us/op SerdeBenchmark.deserialize impressions Avro avgt 3 2.056 ± 1.141 us/op SerdeBenchmark.deserialize metrics JSON avgt 3 5.504 ± 1.535 us/op SerdeBenchmark.deserialize metrics Avro avgt 3 7.564 ± 3.445 us/op SerdeBenchmark.serialize impressions JSON avgt 3 0.556 ± 0.085 us/op SerdeBenchmark.serialize impressions Avro avgt 3 1.420 ± 3.746 us/op SerdeBenchmark.serialize metrics JSON avgt 3 2.909 ± 1.162 us/op SerdeBenchmark.serialize metrics Avro avgt 3 5.076 ± 0.383 us/op ``` Latest benchmark results (no statistically significant change): ``` Benchmark (schemaName) (serializationFormat) Mode Cnt Score Error Units SerdeBenchmark.deserialize impressions JSON avgt 3 1.191 ± 0.520 us/op SerdeBenchmark.deserialize impressions Avro avgt 3 2.032 ± 0.512 us/op SerdeBenchmark.deserialize metrics JSON avgt 3 5.324 ± 1.457 us/op SerdeBenchmark.deserialize metrics Avro avgt 3 7.604 ± 2.389 us/op SerdeBenchmark.serialize impressions JSON avgt 3 0.589 ± 0.027 us/op SerdeBenchmark.serialize impressions Avro avgt 3 1.307 ± 0.147 us/op SerdeBenchmark.serialize metrics JSON avgt 3 2.752 ± 0.583 us/op SerdeBenchmark.serialize metrics Avro avgt 3 5.005 ± 0.780 us/op ```
- Loading branch information
1 parent
c57bdb1
commit d037f26
Showing
29 changed files
with
801 additions
and
1,042 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.