Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink/(cdc): Support multi-table DDL dispatching for MQ sink #5329

Merged
merged 26 commits into from
May 10, 2022
Merged

sink/(cdc): Support multi-table DDL dispatching for MQ sink #5329

merged 26 commits into from
May 10, 2022

Conversation

zhaoxinyu
Copy link
Contributor

@zhaoxinyu zhaoxinyu commented May 5, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #4423

What is changed and how it works?

For the DDLs involving more than one table such as RENAME TABLE, DROP TABLE and DROP VIEW, TiCDC will dispatch the DDL events to the corresponding Kafka topics respectively if the table-level topic dispatch rules are configured.

But there exist some subtle difference between RENAME TABLE and the other two DROP statements.

  • For RENAME TABLE statement, only one DDL job will be generated, so we must parse the DDL events from that job.
  • For DROP TABLE and DROP VIEW statements, multiple DDL jobs containing different table/view info will be generated for each statement. But the query in those jobs are identical to each other. Thus we must rebuild the correct query.

Check List

Tests

  • Unit test
  • Integration test

Release note 

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented May 5, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • asddongmen
  • overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the release-note-none Denotes a PR that doesn't merit a release note. label May 5, 2022
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 5, 2022
@zhaoxinyu zhaoxinyu requested review from Rustin170506 and maxshuang and removed request for 3AceShowHand May 5, 2022 04:13
@ti-chi-bot ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 5, 2022
Rustin170506
Rustin170506 reviewed May 5, 2022
View reviewed changes
Copy link
Member

@Rustin170506 Rustin170506 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What a big PR, is it possible for us to split the drop table and rename table? Also could you please resolve conflicts before continuing? Thanks!

@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 5, 2022
@zhaoxinyu
Copy link
Contributor Author

What a big PR, is it possible for us to split the drop table and rename table? Also could you please resolve conflicts before continuing? Thanks!

The conflicts have been resolved. This PR have bunch of ut modification. But the core logic modification is small.

@zhaoxinyu zhaoxinyu requested a review from asddongmen May 5, 2022 07:33
@codecov-commenter
Copy link

codecov-commenter commented May 5, 2022 •
edited
Loading

Codecov Report

Merging #5329 (a90db0a) into master (687e248) will increase coverage by 0.0994%.
The diff coverage is 54.8977%.

Flag Coverage Δ
cdc 61.3209% <54.8977%> (+0.7222%) ⬆️
dm 52.1744% <ø> (-0.2951%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #5329        +/-   ##
================================================
+ Coverage   56.1240%   56.2234%   +0.0994%     
================================================
  Files           522        529         +7     
  Lines         65325      69809      +4484     
================================================
+ Hits          36663      39249      +2586     
- Misses        25094      26815      +1721     
- Partials       3568       3745       +177

overvenus
overvenus reviewed May 5, 2022
View reviewed changes
cdc/model/sink.go Outdated Show resolved Hide resolved
cdc/model/sink.go Show resolved Hide resolved
cdc/owner/changefeed.go Outdated Show resolved Hide resolved
@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 5, 2022
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 5, 2022
@zhaoxinyu
Copy link
Contributor Author

/run-all-tests

@zhaoxinyu
Copy link
Contributor Author

/run-all-tests

@zhaoxinyu
Copy link
Contributor Author

/run-all-tests

2 similar comments
@zhaoxinyu
Copy link
Contributor Author

/run-all-tests

@zhaoxinyu
Copy link
Contributor Author

/run-all-tests

@zhaoxinyu
Copy link
Contributor Author

/run-kafka-integration-test
/run-integration-test

@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label May 9, 2022
@zhaoxinyu
Copy link
Contributor Author

/run-all-tests

1 similar comment
@zhaoxinyu
Copy link
Contributor Author

/run-all-tests

@zhaoxinyu
Copy link
Contributor Author

/run-kafka-integration-test

@zhaoxinyu
Copy link
Contributor Author

/run-all-tests

@asddongmen
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: b35140f

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 10, 2022
@ti-chi-bot ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label May 10, 2022
@zhaoxinyu
Copy link
Contributor Author

/run-all-tests

@zhaoxinyu
Copy link
Contributor Author

/run-verify

@asddongmen
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: a90db0a

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label May 10, 2022
@zhaoxinyu
Copy link
Contributor Author

/run-verify

@ti-chi-bot ti-chi-bot merged commit c31ba81 into pingcap:master May 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@3AceShowHand 3AceShowHand 3AceShowHand left review comments

@Rustin170506 Rustin170506 Rustin170506 left review comments

@nongfushanquan nongfushanquan nongfushanquan left review comments

@overvenus overvenus overvenus approved these changes

@asddongmen asddongmen asddongmen approved these changes

@maxshuang maxshuang Awaiting requested review from maxshuang

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@zhaoxinyu @ti-chi-bot @codecov-commenter @asddongmen @overvenus @3AceShowHand @Rustin170506 @nongfushanquan