schedule: support patrol region concurrency #8094

lhy1024 · 2024-04-18T08:29:21Z

What problem does this PR solve?

Issue Number: Close #7963 #7706

What is changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Release note

None.

ti-chi-bot · 2024-04-18T08:29:23Z

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2024-04-18T08:29:23Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: lhy1024 <[email protected]>

nolouch

overall lgtm, is this PR ready?

lhy1024 · 2024-05-14T03:32:03Z

overall lgtm, is this PR ready?

I am preparing some tests for different scenarios

Signed-off-by: lhy1024 <[email protected]>

lhy1024 · 2024-05-24T02:24:19Z

pkg/schedule/config/config.go

@@ -63,6 +63,8 @@ const (
 	defaultRegionScoreFormulaVersion = "v2"
 	defaultLeaderSchedulePolicy      = "count"
 	defaultStoreLimitVersion         = "v1"
+	defaultPatrolRegionConcurrency   = 1
+	defaultPatrolRegionBatchLimit    = 128


Maybe we can use max(128,region_count/1024)

pkg/schedule/operator/operator_controller.go

Signed-off-by: lhy1024 <[email protected]>

codecov · 2024-05-24T04:04:30Z

Codecov Report

Attention: Patch coverage is 85.26316% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 77.36%. Comparing base (4cd42b3) to head (ab9ef1e).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8094      +/-   ##
==========================================
+ Coverage   77.29%   77.36%   +0.07%     
==========================================
  Files         471      471              
  Lines       61445    61515      +70     
==========================================
+ Hits        47491    47590      +99     
+ Misses      10395    10362      -33     
- Partials     3559     3563       +4

Flag	Coverage Δ
unittests	`77.36% <85.26%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: lhy1024 <[email protected]>

bufferflies · 2024-09-24T02:50:06Z

pkg/schedule/checker/checker_controller.go

+		// Stop the old workers and start the new workers.
+		c.patrolRegionContext.workersCancel()
+		c.patrolRegionContext.wg.Wait()
+		c.patrolRegionContext.workersCtx, c.patrolRegionContext.workersCancel = context.WithCancel(c.ctx)


Can we adjust the workers more gracefully? For example, if the new worker count is more than the current workers, we can scale out more wroker and no need to build all workers.

I'm not sure it's necessary, and generally speaking we don't change this configuration very often.

I'm afraid that it maybe wait some time to stop all and start all wokers.

pkg/schedule/checker/checker_controller.go

Signed-off-by: lhy1024 <[email protected]>

lhy1024 · 2024-10-23T02:34:50Z

PTAL @bufferflies @rleungx

bufferflies · 2024-10-25T04:02:46Z

pkg/schedule/config/config.go

@@ -67,6 +67,9 @@ const (
 	defaultRegionScoreFormulaVersion = "v2"
 	defaultLeaderSchedulePolicy      = "count"
 	defaultStoreLimitVersion         = "v1"
+	defaultPatrolRegionWorkerCount   = 1
+	maxPatrolRegionWorkerCount       = 8


Maybe it too small and not be changed？how about using the core num as the max limit？

Current tests show that 8 is enough, if needed in the future I think it can be increased or core num can be used.

bufferflies · 2024-10-29T02:04:28Z

rest lgtm

ti-chi-bot · 2024-10-29T06:56:54Z

[LGTM Timeline notifier]

Timeline:

2024-06-13 06:42:23.528767754 +0000 UTC m=+619697.582079678: ☑️ agreed by JmPotato.
2024-06-13 06:42:55.992737053 +0000 UTC m=+619730.046048978: ✖️🔁 reset by JmPotato.
2024-09-25 08:50:25.320316515 +0000 UTC m=+1642295.060740453: ☑️ agreed by okJiang.
2024-10-29 06:56:53.983332403 +0000 UTC m=+332926.822487934: ☑️ agreed by bufferflies.

Signed-off-by: lhy1024 <[email protected]>

lhy1024 · 2024-10-30T10:34:09Z

/test pull-integration-realcluster-test

ti-chi-bot · 2024-10-30T10:41:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bufferflies, niubell, nolouch, okJiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [bufferflies,nolouch]
~~pkg/mcs/scheduling/server/config/OWNERS~~ [niubell]
~~pkg/schedule/config/OWNERS~~ [niubell]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. labels Apr 18, 2024

ti-chi-bot bot requested review from JmPotato and rleungx April 18, 2024 08:29

ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 18, 2024

checker: add patrol region concurrency

d1f4b8a

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the patrol-concurrency branch from 4c59016 to d1f4b8a Compare April 22, 2024 09:52

lhy1024 added 3 commits April 22, 2024 18:21

speedup drain

6d0fbd4

Signed-off-by: lhy1024 <[email protected]>

fix config

9b43d61

Signed-off-by: lhy1024 <[email protected]>

make config

b2b4f39

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the patrol-concurrency branch from 2e61d30 to 948dc77 Compare May 9, 2024 15:33

nolouch reviewed May 14, 2024

View reviewed changes

lhy1024 added 4 commits May 16, 2024 19:54

update

cb285f6

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency-2

2e14405

Signed-off-by: lhy1024 <[email protected]>

fix race

351ef5c

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency-2

c198b08

lhy1024 force-pushed the patrol-concurrency branch from 948dc77 to c198b08 Compare May 23, 2024 17:56

lhy1024 changed the title ~~Patrol concurrency~~ schedule: support patrol region concurrency May 23, 2024

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label May 23, 2024

lhy1024 marked this pull request as ready for review May 23, 2024 17:58

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 23, 2024

lhy1024 commented May 24, 2024

View reviewed changes

pkg/schedule/operator/operator_controller.go Outdated Show resolved Hide resolved

fix test

a0ec33d

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the patrol-concurrency branch from e71f635 to a0ec33d Compare May 24, 2024 03:56

address comments

8cd8825

Signed-off-by: lhy1024 <[email protected]>

bufferflies reviewed Sep 24, 2024

View reviewed changes

pkg/schedule/checker/checker_controller.go Show resolved Hide resolved

lhy1024 added 4 commits September 24, 2024 22:00

address comments

bcd5018

Signed-off-by: lhy1024 <[email protected]>

Merge branch 'master' of github.com:tikv/pd into patrol-concurrency

acb4244

avoid potential data race

457da3d

Signed-off-by: lhy1024 <[email protected]>

address comments: remove sleep in failpoint

64abc3c

Signed-off-by: lhy1024 <[email protected]>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 25, 2024

okJiang approved these changes Sep 25, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 25, 2024

fix lint

e570e3c

Signed-off-by: lhy1024 <[email protected]>

bufferflies reviewed Oct 25, 2024

View reviewed changes

bufferflies approved these changes Oct 29, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Oct 29, 2024

Merge branch 'master' into patrol-concurrency

74d4fdc

nolouch approved these changes Oct 29, 2024

View reviewed changes

ti-chi-bot bot and others added 2 commits October 29, 2024 08:31

Merge branch 'master' into patrol-concurrency

aeabc52

fix lint and make test stable

7e5813d

Signed-off-by: lhy1024 <[email protected]>

lhy1024 force-pushed the patrol-concurrency branch from 02bfc3f to 7e5813d Compare October 30, 2024 09:58

Merge branch 'master' into patrol-concurrency

240f902

niubell approved these changes Oct 30, 2024

View reviewed changes

ti-chi-bot bot added the approved label Oct 30, 2024

ti-chi-bot bot merged commit 649393a into tikv:master Oct 30, 2024
25 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

schedule: support patrol region concurrency #8094

schedule: support patrol region concurrency #8094

lhy1024 commented Apr 18, 2024 •

edited

Loading

ti-chi-bot bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

nolouch left a comment

lhy1024 commented May 14, 2024

lhy1024 May 24, 2024

codecov bot commented May 24, 2024 •

edited

Loading

bufferflies Sep 24, 2024

lhy1024 Sep 24, 2024

bufferflies Sep 24, 2024

lhy1024 commented Oct 23, 2024

bufferflies Oct 25, 2024

lhy1024 Oct 28, 2024

bufferflies Oct 29, 2024

bufferflies commented Oct 29, 2024

ti-chi-bot bot commented Oct 29, 2024

lhy1024 commented Oct 30, 2024

ti-chi-bot bot commented Oct 30, 2024

schedule: support patrol region concurrency #8094

schedule: support patrol region concurrency #8094

Conversation

lhy1024 commented Apr 18, 2024 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Apr 18, 2024

ti-chi-bot bot commented Apr 18, 2024

nolouch left a comment

Choose a reason for hiding this comment

lhy1024 commented May 14, 2024

lhy1024 May 24, 2024

Choose a reason for hiding this comment

codecov bot commented May 24, 2024 • edited Loading

Codecov Report

bufferflies Sep 24, 2024

Choose a reason for hiding this comment

lhy1024 Sep 24, 2024

Choose a reason for hiding this comment

bufferflies Sep 24, 2024

Choose a reason for hiding this comment

lhy1024 commented Oct 23, 2024

bufferflies Oct 25, 2024

Choose a reason for hiding this comment

lhy1024 Oct 28, 2024

Choose a reason for hiding this comment

bufferflies Oct 29, 2024

Choose a reason for hiding this comment

bufferflies commented Oct 29, 2024

ti-chi-bot bot commented Oct 29, 2024

[LGTM Timeline notifier]

lhy1024 commented Oct 30, 2024

ti-chi-bot bot commented Oct 30, 2024

lhy1024 commented Apr 18, 2024 •

edited

Loading

codecov bot commented May 24, 2024 •

edited

Loading