Better Error Message for Schema Evolution Failure

[agavra]

Today we check for avro schema compatibility when we create a new schema registry subject:

https://github.com/confluentinc/ksql/blob/master/ksql-engine/src/main/java/io/confluent/ksql/engine/EngineExecutor.java#L293

This has a lot of avro-specific code that we should make more generic. It probably also doesn't belong when making the sink DDL but when we create the topic.

Either way, we should make sure that this evolution check is there for both JSON and PROTOBUF when using schema registry.

Also see #4219, which will hopefully fix this issue as well.

[agavra]

Schema evolution is now fixed with #4717 but the error message isn't very good...

ksql> CREATE STREAM foo_one (id VARCHAR) WITH (kafka_topic='foo', value_format='PROTOBUF', partitions=1);

 Message
----------------
 Stream created
----------------
ksql> CREATE STREAM foo_two (id INT) WITH (kafka_topic='foo', value_format='PROTOBUF');
Could not register schema for topic.
ksql> CREATE STREAM foo_two (id VARCHAR, col2 VARCHAR) WITH (kafka_topic='foo', value_format='PROTOBUF');

 Message
----------------
 Stream created
----------------
ksql>

There are two TODOs to come out of this:

• better error message for when schema registration fails for compatibility reasons
• remove the old AvroUtil schema registry checks as they are no longer needed

Furthermore, the AVRO path doesn't even get to the part that it used to with AvroUtil because it gets cut short earlier - so the error message has some level of regression.

[rmoff]

Can we look to fix this error message soon? It's not very helpful, and requires the user to go to the Schema Registry log to see the cause, e.g.:

io.confluent.kafka.schemaregistry.rest.exceptions.RestIncompatibleSchemaException: Schema being registered is incompatible with an earlier schema for subject "livecyclehireupdates_protobuf_02-value"

[vcrfxia]

Hey @rmoff , what ksqlDB version were you running in October when you reported that this error message still hadn't been fixed? I tried replicating the issue just now but it appears to have been fixed:

ksql> create stream foo (id int) with (kafka_topic='foo', value_format='avro', partitions=1);

 Message
----------------
 Stream created
----------------
ksql> create stream foo2 with (kafka_topic='foo') as select 'string' as id from foo;
Could not register schema for topic: Schema being registered is incompatible with an earlier schema; error code: 409

Note that I had to try reproducing with a CSAS statement rather than another CS statement since ksqlDB no longer registers new schemas for CS statements if there's already an existing one in Schema Registry.

There are two TODOs to come out of this:

better error message for when schema registration fails for compatibility reasons
remove the old AvroUtil schema registry checks as they are no longer needed

The second TODO has already been completed in #6452, so unless I've missed something here this issue can be closed.

[rmoff]

@vcrfxia good question :D I should have included that.
If you can't reproduce it then let's close for now and if I encounter it again I'll reopen with info that's more helpful this time.