sink(ticdc): add GetActiveTopics support for event router #4743

Rustin170506 · 2022-03-02T09:58:33Z

What problem does this PR solve?

Issue Number: ref #4423

What is changed and how it works?

add GetActiveTopics support for event router. It will return the topic corresponding to each table (de-duplicated).
pass table names to EmitCheckpointTs. It will also pass the tables in the schema that need to be synchronized at that time.

Check List

Tests

Unit test

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change

Side effects

None

Related changes

None

Release note


None

ti-chi-bot · 2022-03-02T09:58:34Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

amyangfei
zhaoxinyu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov-commenter · 2022-03-03T06:33:36Z

Codecov Report

Merging #4743 (89fe2db) into master (9607554) will decrease coverage by 0.0829%.
The diff coverage is 52.4492%.

Flag	Coverage Δ
cdc	`59.7950% <52.4492%> (-0.1272%)`	⬇️
dm	`52.0460% <ø> (+0.0171%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #4743        +/-   ##
================================================
- Coverage   55.6402%   55.5572%   -0.0830%     
================================================
  Files           494        521        +27     
  Lines         61283      64393      +3110     
================================================
+ Hits          34098      35775      +1677     
- Misses        23750      25104      +1354     
- Partials       3435       3514        +79

Rustin170506 · 2022-03-03T08:18:37Z

/run-all-tests

…r sink

Rustin170506 · 2022-03-04T09:34:32Z

cdc/sink/dispatcher/event_router.go

 		partitionDispatcher partition.Dispatcher
 		topicDispatcher     topic.Dispatcher
 		filter.Filter
 	}
 }

+// NewEventRouter creates a new EventRouter
+func NewEventRouter(cfg *config.ReplicaConfig, defaultTopic string) (*EventRouter, error) {


The check for defaultTopic is insufficient here, but it's not related to this PR, so I'll fix it in the next PR.

Rustin170506 · 2022-03-04T10:04:02Z

/run-integration-test /tikv=pr/12050
/run-kafka-integration-test /tikv=pr/12050

Rustin170506 · 2022-03-04T10:20:35Z

/run-leak-test
/run-verify

amyangfei

Besides how does passing all table names solve the table missing problem

cdc/owner/schema.go

cdc/owner/schema_test.go

Rustin170506 · 2022-03-07T03:38:03Z

Besides how does passing all table names solve the table missing problem

I think we just need to maintain all the table information that corresponds at that time? It is in the tick before the processing of the ddl, so can we assume that the tables that need to be synchronized at the current moment are those in AllTableNames?

cdc/owner/ddl_sink.go

cdc/owner/schema.go

cdc/sink/dispatcher/event_router.go

zhaoxinyu · 2022-03-07T09:21:37Z

/LGTM

Rustin170506

Self check

Rustin170506 · 2022-03-07T09:33:00Z

/hold

/cc @amyangfei

Please take a look again. Thanks!

cdc/owner/changefeed.go

Rustin170506 · 2022-03-07T11:03:56Z

/run-all-tests

Rustin170506 · 2022-03-07T11:27:44Z

/run-kafka-integration-test

Rustin170506 · 2022-03-07T11:51:51Z

/unhold

/merge

ti-chi-bot · 2022-03-07T11:51:53Z

This pull request has been accepted and is ready to merge.

Commit hash: d65308c

ti-chi-bot · 2022-03-07T12:23:59Z

@hi-rustin: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Rustin170506 · 2022-03-07T12:44:26Z

/run-verify

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 2, 2022

Rustin170506 force-pushed the rustin-patch-sink-hooks branch 2 times, most recently from 5492128 to bbf7ee4 Compare March 3, 2022 06:07

Rustin170506 force-pushed the rustin-patch-sink-hooks branch 3 times, most recently from a6f10c4 to aeb260f Compare March 3, 2022 11:56

Rustin170506 changed the title ~~sink(ticdc): add OnTableSinkCreated and OnTableSinkDestroyed hooks for sink~~ sink(ticdc): add GetActiveTopics support for event router Mar 4, 2022

Rustin170506 marked this pull request as ready for review March 4, 2022 09:16

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 4, 2022

sink(ticdc): add OnTableSinkCreated and OnTableSinkDestroyed hooks fo…

ba03ecf

…r sink

Rustin170506 force-pushed the rustin-patch-sink-hooks branch from b454abf to ba03ecf Compare March 4, 2022 09:22

Rustin170506 commented Mar 4, 2022

View reviewed changes

Rustin170506 requested review from zhaoxinyu and amyangfei March 4, 2022 09:38

Merge branch 'master' into rustin-patch-sink-hooks

24e6407

amyangfei reviewed Mar 7, 2022

View reviewed changes

cdc/owner/schema.go Outdated Show resolved Hide resolved

cdc/owner/schema.go Outdated Show resolved Hide resolved

cdc/owner/schema_test.go Outdated Show resolved Hide resolved

sink(ticdc): address comments

89fe2db

Rustin170506 requested a review from amyangfei March 7, 2022 03:38

amyangfei approved these changes Mar 7, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 7, 2022

zhaoxinyu reviewed Mar 7, 2022

View reviewed changes

cdc/owner/ddl_sink.go Show resolved Hide resolved

zhaoxinyu reviewed Mar 7, 2022

View reviewed changes

cdc/owner/schema.go Outdated Show resolved Hide resolved

zhaoxinyu reviewed Mar 7, 2022

View reviewed changes

cdc/sink/dispatcher/event_router.go Show resolved Hide resolved

sink(ticdc): use current table names

a2c1848

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 7, 2022

Rustin170506 requested a review from zhaoxinyu March 7, 2022 08:56

Rustin170506 added 2 commits March 7, 2022 17:17

sink(ticdc): add log

20a9a41

sink(ticdc): remove comment

e5ba10a

zhaoxinyu approved these changes Mar 7, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 7, 2022

Rustin170506 commented Mar 7, 2022

View reviewed changes

ti-chi-bot requested a review from amyangfei March 7, 2022 09:33

ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 7, 2022

amyangfei reviewed Mar 7, 2022

View reviewed changes

cdc/owner/changefeed.go Show resolved Hide resolved

Merge branch 'master' into rustin-patch-sink-hooks

d65308c

amyangfei approved these changes Mar 7, 2022

View reviewed changes

ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 7, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 7, 2022

ti-chi-bot added 2 commits March 7, 2022 19:52

Merge branch 'master' into rustin-patch-sink-hooks

ed0378c

Merge branch 'master' into rustin-patch-sink-hooks

1c70957

ti-chi-bot merged commit 54b129d into pingcap:master Mar 7, 2022

Rustin170506 deleted the rustin-patch-sink-hooks branch March 7, 2022 13:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sink(ticdc): add GetActiveTopics support for event router #4743

sink(ticdc): add GetActiveTopics support for event router #4743

Rustin170506 commented Mar 2, 2022 •

edited

Loading

ti-chi-bot commented Mar 2, 2022 •

edited

Loading

codecov-commenter commented Mar 3, 2022 •

edited

Loading

Rustin170506 commented Mar 3, 2022

Rustin170506 Mar 4, 2022

Rustin170506 commented Mar 4, 2022

Rustin170506 commented Mar 4, 2022

amyangfei left a comment

Rustin170506 commented Mar 7, 2022

zhaoxinyu commented Mar 7, 2022

Rustin170506 left a comment

Rustin170506 commented Mar 7, 2022

Rustin170506 commented Mar 7, 2022

Rustin170506 commented Mar 7, 2022

Rustin170506 commented Mar 7, 2022

ti-chi-bot commented Mar 7, 2022

ti-chi-bot commented Mar 7, 2022

Rustin170506 commented Mar 7, 2022

sink(ticdc): add GetActiveTopics support for event router #4743

sink(ticdc): add GetActiveTopics support for event router #4743

Conversation

Rustin170506 commented Mar 2, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Mar 2, 2022 • edited Loading

codecov-commenter commented Mar 3, 2022 • edited Loading

Codecov Report

Rustin170506 commented Mar 3, 2022

Rustin170506 Mar 4, 2022

Choose a reason for hiding this comment

Rustin170506 commented Mar 4, 2022

Rustin170506 commented Mar 4, 2022

amyangfei left a comment

Choose a reason for hiding this comment

Rustin170506 commented Mar 7, 2022

zhaoxinyu commented Mar 7, 2022

Rustin170506 left a comment

Choose a reason for hiding this comment

Rustin170506 commented Mar 7, 2022

Rustin170506 commented Mar 7, 2022

Rustin170506 commented Mar 7, 2022

Rustin170506 commented Mar 7, 2022

ti-chi-bot commented Mar 7, 2022

ti-chi-bot commented Mar 7, 2022

Rustin170506 commented Mar 7, 2022

Rustin170506 commented Mar 2, 2022 •

edited

Loading

ti-chi-bot commented Mar 2, 2022 •

edited

Loading

codecov-commenter commented Mar 3, 2022 •

edited

Loading