lightning: split and scatter regions in batches (#33625) #34259
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: ti-srebot <[email protected]>
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
|
/run-all-tests |
ti-chi-bot
added
do-not-merge/release-note-label-needed
4 tasks
ti-srebot
added
component/lightning
|
@gozssky you're already a collaborator in bot's repo. |
D3Hunter
approved these changes
Apr 27, 2022
ti-chi-bot
added
status/LGT2
|
closed for DMR version |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cherry-pick #33625 to release-6.0
You can switch your code base to this Pull Request by using git-extras:
# In tidb repo: git pr https://github.com/pingcap/tidb/pull/34259After apply modifications, you can push your change to this PR via:
What problem does this PR solve?
Issue Number: close #33618
Problem Summary:
Region scatter may timeout when split and scatter many regions at a time.
What is changed and how it works?
Add a batch limit for split and scatter regions. In a batch, lightning processes up to 4096 ranges. After all ranges in batch have been processed completely, lightning can process the next batch.
Check List
Tests
I prepared about 1.8T data for a table which contains only one clustered primary key. TiDB cluster have ten TiKV nodes, each node have 200% cpu limit and 16GiB memory limit. Then I used lightning to import these data to TiDB cluster and observed the scatter operator status in grafana dashboard. To reproduce huge amount of regions, I set
region-split-sizeto 8MiB. Finally, about 260k regions were created.Master
Many scatter operator is timeout.
This PR
There is no operator timeout after this PR.
Release note