Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

frontier(ticdc): fast check if region is split or merged when forwarding resolvedTs #7280

Merged
merged 32 commits into from
Oct 9, 2022
Merged

frontier(ticdc): fast check if region is split or merged when forwarding resolvedTs #7280

merged 32 commits into from
Oct 9, 2022

Conversation

sdojjy
Copy link
Member

@sdojjy sdojjy commented Sep 30, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: close #7281

What is changed and how it works?

using map cached all region spans, when forwarding a span, check if it is split or merged by comparing the span start, span end, and region ID.

server CPU and memory:
image

profile before:
image

profile after
image

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

improve the performance of the frointier

sdojjy and others added 19 commits September 22, 2022 12:06
@sdojjy
add log rate limit for table actor
85d1197
@ti-chi-bot
Merge branch 'master' into add-log-rate-limit-for-table-actro
0eac0e7
@ti-chi-bot
Merge branch 'master' into add-log-rate-limit-for-table-actro
6701809
@ti-chi-bot
Merge branch 'master' into add-log-rate-limit-for-table-actro
5670c40
@ti-chi-bot
Merge branch 'master' into add-log-rate-limit-for-table-actro
97db214
@ti-chi-bot
Merge branch 'master' into add-log-rate-limit-for-table-actro
d8fff7a
@sdojjy
Merge branch 'master' into add-log-rate-limit-for-table-actro
72fa8a6
@sdojjy
add rate limiter for resolve lock
f9e9107
@sdojjy
improve frontier
19a00da
@sdojjy
improve frontier
c33dc39
@sdojjy
improve frontier
9ef6307
@sdojjy
add metrics
69a013c
@sdojjy
add ut
d837620
@sdojjy
add ut
39e2938
@sdojjy
add ut
253c3c0
@sdojjy
add ut
89338c2
@sdojjy
add ut
0a0e268
@sdojjy
Merge remote-tracking branch 'upstream/master' into frointier-improve
d52391b 
# Conflicts:
#	cdc/kv/region_worker.go
#	cdc/processor/pipeline/table_actor.go
@sdojjy
fix lint
02ec537
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Sep 30, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • 3AceShowHand
  • asddongmen

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 30, 2022
@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 30, 2022
@sdojjy
Copy link
Member Author

sdojjy commented Sep 30, 2022

/run-all-tests

@amyangfei amyangfei added the area/ticdc Issues or PRs related to TiCDC. label Sep 30, 2022
@sdojjy sdojjy changed the title frontier(ticdc): fast check if region is split or merge when forwarding resolvedTs frontier(ticdc): fast check if region is split or merged when forwarding resolvedTs Oct 8, 2022
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 8, 2022
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 9, 2022
@asddongmen
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 43b9b4f

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Oct 9, 2022
@sdojjy
Copy link
Member Author

sdojjy commented Oct 9, 2022

/merge

@3AceShowHand
Copy link
Contributor

/run-verify-ci

@ti-chi-bot ti-chi-bot merged commit 0f2a253 into pingcap:master Oct 9, 2022
@sdojjy sdojjy added the needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. label Oct 10, 2022
@ti-chi-bot ti-chi-bot mentioned this pull request Oct 10, 2022
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #7303.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Oct 10, 2022
@sdojjy @ti-chi-bot
This is an automated cherry-pick of pingcap#7280
3402751 
Signed-off-by: ti-chi-bot <[email protected]>
@3AceShowHand 3AceShowHand mentioned this pull request Oct 10, 2022
Closed
ti-chi-bot added a commit that referenced this pull request Oct 10, 2022
@ti-chi-bot
frontier(ticdc): fast check if region is split or merged when forward…
79bf46d 
…ing resolvedTs (#7280) (#7303)

close #7281
@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Nov 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nongfushanquan nongfushanquan nongfushanquan left review comments

@asddongmen asddongmen asddongmen approved these changes

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

@amyangfei amyangfei Awaiting requested review from amyangfei

Assignees
No one assigned
Labels
area/ticdc Issues or PRs related to TiCDC. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

fast check if region is split or merged when forwarding resolvedTs
6 participants
@sdojjy @ti-chi-bot @asddongmen @3AceShowHand @nongfushanquan @amyangfei