Handle QualifiedNameReference when selecting stream/table keys #1697
Conversation
|
Is this a fix for #1686 , or just a step toward a fix? |
| SchemaKStream projectedSchemaKStream = initialSchemaKStream.select(projectNameExpressionPairList); | ||
| assertThat( | ||
| projectedSchemaKStream.getKeyField(), | ||
| equalTo(new Field("COL0", 2, Schema.OPTIONAL_INT64_SCHEMA))); |
There was a problem hiding this comment.
Why isn't this just ksqlStream.getKeyField()? That should be the key of the derived stream, right?
There was a problem hiding this comment.
Yeah in this case that's true. I'll change it.
| "CREATE TABLE TEST_TABLE (ID bigint, F1 varchar, F2 bigint) WITH (kafka_topic='right_topic', value_format='{FORMAT}', key='ID');", | ||
| "CREATE TABLE TEST_TABLE_2 (ID bigint, F3 varchar) WITH (kafka_topic='right_topic_2', value_format='{FORMAT}', key='ID');", | ||
| "CREATE TABLE INNER_JOIN WITH (PARTITIONS=1) as SELECT t.id, name, value, f1, f2 FROM test t join TEST_TABLE tt on t.id = tt.id;", | ||
| "CREATE TABLE INNER_JOIN_2 AS SELECT t_id, name, f1, f3 FROM inner_join tt join TEST_TABLE_2 t ON t.id = tt.t_id;" |
There was a problem hiding this comment.
So basically the key field for the join stream is the of the form leftTable_leftKeyField? Is this documented anywhere?
There was a problem hiding this comment.
That's the format for columns selected using the '.' operator.
There was a problem hiding this comment.
ok.. is this documented? Because it is important to do joins downstream, isn't it? If it isn't, we should file an issue to update the documentation.
rodesai
force-pushed
the
fix_key_selection_5.0.x
branch
from
d86ece7
to
df0fdea
Compare
| @@ -312,6 +312,27 @@ | |||
| {"topic": "INNER_JOIN", "key": 0, "value": {"T_ID": 0, "NAME": "foo", "VALUE": 100, "F1": "zero", "F2": 0}, "timestamp": 10000}, | |||
| {"topic": "INNER_JOIN", "key": 0, "value": {"T_ID": 0, "NAME": "bar", "VALUE": 99, "F1": "a", "F2": 10}, "timestamp": 15000} | |||
| ] | |||
| }, | |||
| { | |||
| "name": "multi-way join", | |||
There was a problem hiding this comment.
Maybe rename this since multi-way join would resemble multiple joins in one query, like 3-way join. How about join-pipeline?
rodesai
force-pushed
the
fix_key_selection_5.0.x
branch
from
df0fdea
to
f5ec518
Compare
SchemaKStream.select may get a QualifiedNameReference as the expression in a select call. The logic that looks for and preserves the key field in the resulting SchemaKStream needs to handle expressions of this type. It also needs to handle cases where the key field name is fully qualified, as happens with joins.
rodesai
force-pushed
the
fix_key_selection_5.0.x
branch
from
f5ec518
to
15ab445
Compare
Description
SchemaKStream.select may get a QualifiedNameReference as the expression
in a select call. The logic that looks for and preserves the key field in
the resulting SchemaKStream needs to handle expressions of this type. It
also needs to handle cases where the key field name is fully qualified, as
happens with joins.
Fixes #1686. Fixes #1550 .
Testing done
Describe the testing stategy. Unit and integration tests are expected for any behavior changes.
Reviewer checklist