Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

capture(cdc), tcpserver: integrate P2P with server & capture #3932

Merged
merged 15 commits into from
Dec 23, 2021
Merged

capture(cdc), tcpserver: integrate P2P with server & capture #3932

merged 15 commits into from
Dec 23, 2021

Conversation

liuzix
Copy link
Contributor

@liuzix liuzix commented Dec 17, 2021

What problem does this PR solve?

  • Prepares basic dependencies for the peer-messaging based scheduler to work.

What is changed and how it works?

  • Uses pkg/tcpserver to listen to the port.
  • Runs the gRPC server.
  • Runs the MessageServer.
  • Maintains the MessageServer, MessageRouter and ServerWrapper in Capture.
  • Injects MessageServer and MessageRouter into context so that the Owner and Processor can depend on them.
  • Calls SetOnCaptureAdded and SetOnCaptureRemoved on the global state reactor so that the MessageRouter can be informed of capture membership changes.

Check List

Tests

  • Unit test
  • Integration test

Release note 

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 17, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • 3AceShowHand
  • amyangfei

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 17, 2021
@liuzix liuzix changed the title capture: integrate P2P with server & capture capture(cdc): integrate P2P with server & capture Dec 17, 2021
@liuzix liuzix added status/ptal Could you please take a look? area/ticdc Issues or PRs related to TiCDC. labels Dec 17, 2021
@liuzix liuzix requested review from asddongmen, overvenus and amyangfei and removed request for asddongmen December 17, 2021 04:36
@liuzix liuzix added this to the v5.4.0 milestone Dec 17, 2021
@liuzix
Copy link
Contributor Author

liuzix commented Dec 17, 2021

/run-integration-tests

1 similar comment
@liuzix
Copy link
Contributor Author

liuzix commented Dec 17, 2021

/run-integration-tests

@liuzix
Copy link
Contributor Author

liuzix commented Dec 17, 2021

/run-leak-tests

@codecov-commenter
Copy link

codecov-commenter commented Dec 17, 2021 •
edited
Loading

Codecov Report

Merging #3932 (5ababae) into master (3873d39) will decrease coverage by 1.9299%.
The diff coverage is 52.0179%.

Flag Coverage Δ
cdc 57.9914% <54.1009%> (-0.2452%) ⬇️
dm 52.7190% <49.2723%> (-3.3156%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #3932        +/-   ##
================================================
- Coverage   57.0741%   55.1442%   -1.9300%     
================================================
  Files           478        484         +6     
  Lines         56551      59183      +2632     
================================================
+ Hits          32276      32636       +360     
- Misses        20978      23247      +2269     
- Partials       3297       3300         +3

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 18, 2021
@liuzix liuzix force-pushed the new-scheduler-capture-server-integration branch from 5ababae to 63db282 Compare December 20, 2021 05:32
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 20, 2021
@liuzix
Copy link
Contributor Author

liuzix commented Dec 20, 2021

/run-integration-tests

1 similar comment
@liuzix
Copy link
Contributor Author

liuzix commented Dec 20, 2021

/run-integration-tests

@liuzix
Copy link
Contributor Author

liuzix commented Dec 20, 2021

/run-verify

overvenus
overvenus reviewed Dec 20, 2021
View reviewed changes
Copy link
Member

@overvenus overvenus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

cdc/capture/capture.go
Comment on lines +183 to +184
c.MessageRouter.Close()
c.MessageRouter.Wait()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we wait in Close?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating Close and Wait in the design of the interface is for two reasons:

  1. To provide a finer-grained control over the timing of execution to achieve better unit-test coverage.
  2. To avoid introducing deadlock bugs accidentally in the future. The pattern of close-wait-close-wait instead of close-close-wait-wait has created deadlock bugs before.

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 21, 2021
@ti-chi-bot ti-chi-bot removed the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 22, 2021
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 22, 2021
@liuzix
Copy link
Contributor Author

liuzix commented Dec 22, 2021

/run-integration-tests

@liuzix
Copy link
Contributor Author

liuzix commented Dec 22, 2021

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 50a444a

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 22, 2021
@liuzix
Copy link
Contributor Author

liuzix commented Dec 22, 2021

/run-leak-tests

@liuzix
Copy link
Contributor Author

liuzix commented Dec 22, 2021

/run-kafka-integration-test

@liuzix
Copy link
Contributor Author

liuzix commented Dec 23, 2021

/run-integration-tests

@liuzix
Copy link
Contributor Author

liuzix commented Dec 23, 2021

/run-dm-integration-test

2 similar comments
@liuzix
Copy link
Contributor Author

liuzix commented Dec 23, 2021

/run-dm-integration-test

@liuzix
Copy link
Contributor Author

liuzix commented Dec 23, 2021

/run-dm-integration-test

@GMHDBJD
Copy link
Contributor

GMHDBJD commented Dec 23, 2021

test failed due to #4029

@asddongmen
Copy link
Contributor

/run-leak-tests
/run-verify

@ti-chi-bot
Copy link
Member

@liuzix: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit f8412fa into pingcap:master Dec 23, 2021
zhaoxinyu pushed a commit to zhaoxinyu/ticdc that referenced this pull request Dec 29, 2021
@liuzix @zhaoxinyu
capture(cdc), tcpserver: integrate P2P with server & capture (pingcap…
13497c7 
…#3932)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@overvenus overvenus overvenus left review comments

@amyangfei amyangfei amyangfei approved these changes

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

Assignees
No one assigned
Labels
area/ticdc Issues or PRs related to TiCDC. release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. status/ptal Could you please take a look?
Projects
None yet
Milestone
v5.4.0
Development

Successfully merging this pull request may close these issues.
8 participants
@liuzix @ti-chi-bot @codecov-commenter @GMHDBJD @asddongmen @amyangfei @overvenus @3AceShowHand