Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: always use the changelog subject in table state stores #5823

Merged
merged 1 commit into from
Jul 16, 2020

Conversation

agavra
Copy link
Contributor

@agavra agavra commented Jul 14, 2020

fixes #5673
replaces #5812

Description

This commit takes a different approach to fixing the issue described in the two links above; we will hardcode the serializer for source tables to use the changelog topic, and we allow the deserializer to fallback to the source topic if it fails to deserialize using the changelog topic.

This fixes the issue by obviating apache/kafka#8902 - all queries that were created before #5781 was merged will always pass in the changelog topic to their state store serde. This means that:

  1. we will never register a schema to the user's subject unless they use INSERT INTO
  2. we will be able to deserialize events in the state store if the schema ID for the event is in either subject
  3. if we ever encounter an event we could not serialize using the changelog subject, we will register that schema into the changelog subject

Review Guide

  • The entrypoint for this change is in SourceBuilder, where we pass into the Consumed instance a custom Serde that spoofs the topic to always use the changelog topic
  • The spoofed serde is StaticTopicSerde
  • That serde accepts a callback RegisterSchemaCallback to help optimize repeated failed deserialization attempts
  • In order to prevent false positive logging to the processing logger, the LoggingDeserializer is extended to support delaying the exceptional case

Testing done

Reviewer checklist

  • Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
  • Ensure relevant issues are linked (description should include text like "Fixes #")

@agavra agavra marked this pull request as ready for review July 14, 2020 20:04
@agavra agavra requested a review from a team as a code owner July 14, 2020 20:04
@agavra agavra requested review from rodesai and vcrfxia July 14, 2020 20:04
Copy link
Contributor

@rodesai rodesai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unreadable state store when running a query with a source table with a schema-registry schema
2 participants