Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-36184][transform] Fix transform operator swallows schema changes from tables not present in transform rules #3589

Conversation

yuxiqian
Copy link
Contributor

@yuxiqian yuxiqian commented Aug 29, 2024

This closes FLINK-36184.

Currently, tables that are not present in transform blocks should be treated as if there's such a dummy fallback block:

transform:
  - source-table: "\.*.\.*" # capture all unmentioned tables
    projection: "*"         # keep all columns
                            # without filtering any rows

There's a bug in #3557's implementation, where schema change events should be filtered out if there's no wildcard (*) in transform rules. However, it also filters out those tables that are not defined in transform rules, which causes schema change events lost with the following example:

transform:
  - source-table: foo.bar.baz   # Another table that doesn't really matter
    projection: ...

Here, since there's one transform block, TransformOperator will be added into operator chain. Now let's perform some schema change events in another table (like db.table), it will be filtered out since TransformOperator regards it as a asterisk-less table and does not require schema change events.

By checking if a table is transformed or not, we could set hasAsterisk flag map correctly and resolve this problem.

@yuxiqian
Copy link
Contributor Author

@ruanhang1993 PTAL

@yuxiqian yuxiqian changed the title [hotfix][transform] Fix transform operator swallows schema changes from tables not present in transform rules [FLINK-36184][transform] Fix transform operator swallows schema changes from tables not present in transform rules Aug 30, 2024
Copy link
Contributor

@ruanhang1993 ruanhang1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@ruanhang1993 ruanhang1993 merged commit 06fc939 into apache:master Aug 30, 2024
21 checks passed
qiaozongmi pushed a commit to qiaozongmi/flink-cdc that referenced this pull request Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants