add min tso scheduler for each resource group #8072

guo-shaoge · 2023-09-08T05:53:24Z

What problem does this PR solve?

Issue Number: close #7919

Problem Summary:

What is changed and how it works?

For each resource group, there is a min tso scheduler(move active_set/waiting_set/min_query_id/estimiated_threads inside GroupEntry)
For min tso hard limit, there is a global_estimated_threads_usage.
Attach resource_group_name for each min tso metrics

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

Signed-off-by: guo-shaoge <[email protected]>

dbms/src/Flash/Mpp/MinTSOScheduler.h

…trol_part_4

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge · 2023-09-14T03:09:22Z

dbms/src/Flash/Mpp/MinTSOScheduler.h

+        bool updateMinQueryId(const MPPQueryId & query_id, bool retired, const String & msg, LoggerPtr log);
+    };
+
+    void scheduleWaitingQueries(GroupEntry & entry, MPPTaskManager & task_manager, LoggerPtr log);


Will call MinTSOScheduler::scheduleImpl(), so should not make it as member function of GroupEntry.

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge · 2023-09-14T03:50:58Z

/run-all-tests

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge · 2023-09-14T06:06:48Z

dbms/src/Flash/Executor/PipelineExecutor.cpp

@@ -110,11 +110,6 @@ ExecutionResult PipelineExecutor::execute(ResultHandler && result_handler)
    }
    LOG_TRACE(log, "query finish with {}", exec_context.getQueryProfileInfo().toJson());

-    // For read_ru, only report it to GAC for now.
-    LocalAdmissionController::global_instance->consumeResource(


will consume read_ru in next pr

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge · 2023-09-14T08:02:41Z

/run-all-tests

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge · 2023-09-14T09:53:33Z

/run-all-tests

Signed-off-by: guo-shaoge <[email protected]>

dbms/src/Flash/ResourceControl/TokenBucket.cpp

guo-shaoge · 2023-09-15T05:43:09Z

/run-all-tests

Signed-off-by: guo-shaoge <[email protected]>

…sh into resource_control_part_4

guo-shaoge · 2023-09-19T03:29:52Z

/run-all-tests

dbms/src/Flash/Mpp/MinTSOScheduler.cpp

Signed-off-by: guo-shaoge <[email protected]>

…sh into resource_control_part_4

Co-authored-by: SeaRise <[email protected]>

guo-shaoge · 2023-09-19T08:13:54Z

/run-all-tests

Signed-off-by: guo-shaoge <[email protected]>

dbms/src/Flash/Mpp/MinTSOScheduler.cpp

dbms/src/Flash/Mpp/MinTSOScheduler.h

dbms/src/Flash/Mpp/MinTSOScheduler.cpp

…sh into resource_control_part_4

Signed-off-by: guo-shaoge <[email protected]>

SeaRise

Others LGTM

SeaRise · 2023-09-20T04:35:11Z

dbms/src/Flash/ResourceControl/TokenBucket.cpp

-    auto elapsed_second = std::chrono::duration_cast<std::chrono::seconds>(elspased).count();
-    return elapsed_second * fill_rate;
+    auto elapsed_ms = std::chrono::duration_cast<std::chrono::milliseconds>(elspased).count();
+    return static_cast<double>(elapsed_ms * fill_rate) / 1000;


It looks like / 1000 doesn't need to be called multiple times.
fill_rate_ms = fill_rate/1000

Signed-off-by: guo-shaoge <[email protected]>

windtalker

LGTM

ti-chi-bot · 2023-09-20T05:27:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: SeaRise, windtalker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [SeaRise,windtalker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-09-20T05:27:40Z

[LGTM Timeline notifier]

Timeline:

2023-09-20 04:35:42.460631789 +0000 UTC m=+664908.428219824: ☑️ agreed by SeaRise.
2023-09-20 05:27:39.434944519 +0000 UTC m=+668025.402532554: ☑️ agreed by windtalker.

ti-chi-bot · 2023-09-20T05:28:56Z

@guo-shaoge: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

This reverts commit 939a18d. Signed-off-by: guo-shaoge <[email protected]>

refine MinTSOScheduler for resource control

25ef095

Signed-off-by: guo-shaoge <[email protected]>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 8, 2023

guo-shaoge commented Sep 8, 2023

View reviewed changes

dbms/src/Flash/Mpp/MinTSOScheduler.h Outdated Show resolved Hide resolved

guo-shaoge added 3 commits September 14, 2023 10:20

Merge branch 'master' of github.com:pingcap/tiflash into resource_con…

53c15ed

…trol_part_4

refine metric

d7335e2

Signed-off-by: guo-shaoge <[email protected]>

detail -> entry

3b66abd

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge commented Sep 14, 2023

View reviewed changes

guo-shaoge added 2 commits September 14, 2023 11:23

case; refine

8da9c62

Signed-off-by: guo-shaoge <[email protected]>

rename tso metric name

2a78ab9

Signed-off-by: guo-shaoge <[email protected]>

rename metric name

3dd452b

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge commented Sep 14, 2023

View reviewed changes

guo-shaoge added 2 commits September 14, 2023 14:08

fix case

e5cc3a5

Signed-off-by: guo-shaoge <[email protected]>

refine

e77b04e

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge changed the title ~~refine MinTSOScheduler for resource control~~ add min tso scheduler for each resource group Sep 14, 2023

guo-shaoge added 2 commits September 14, 2023 16:27

refine

5aedfbd

Signed-off-by: guo-shaoge <[email protected]>

fix

c51248f

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge requested review from windtalker and SeaRise September 14, 2023 09:50

fmt

fea6f0a

Signed-off-by: guo-shaoge <[email protected]>

SeaRise reviewed Sep 14, 2023

View reviewed changes

dbms/src/Flash/ResourceControl/TokenBucket.cpp Outdated Show resolved Hide resolved

Merge branch 'master' into resource_control_part_4

4c40a00

guo-shaoge added 2 commits September 15, 2023 14:09

unit test for TokenBucket::compact()

e6cf492

Signed-off-by: guo-shaoge <[email protected]>

Merge branch 'resource_control_part_4' of github.com:guo-shaoge/tifla…

3de0125

…sh into resource_control_part_4

guo-shaoge requested a review from SeaRise September 18, 2023 08:07

Merge branch 'master' into resource_control_part_4

8a9313d

SeaRise reviewed Sep 19, 2023

View reviewed changes

dbms/src/Flash/Mpp/MinTSOScheduler.cpp Outdated Show resolved Hide resolved

dbms/src/Flash/Mpp/MinTSOScheduler.cpp Outdated Show resolved Hide resolved

dbms/src/Flash/Mpp/MinTSOScheduler.cpp Show resolved Hide resolved

guo-shaoge and others added 4 commits September 19, 2023 15:50

refine compact()

c7f0d53

Signed-off-by: guo-shaoge <[email protected]>

Merge branch 'resource_control_part_4' of github.com:guo-shaoge/tifla…

c159c8a

…sh into resource_control_part_4

Update dbms/src/Flash/Mpp/MinTSOScheduler.cpp

26efb8b

Co-authored-by: SeaRise <[email protected]>

Update dbms/src/Flash/Mpp/MinTSOScheduler.cpp

5a7f854

Co-authored-by: SeaRise <[email protected]>

RUNTIME_CHECK -> assert

acff759

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge requested a review from SeaRise September 19, 2023 14:05

windtalker reviewed Sep 20, 2023

View reviewed changes

dbms/src/Flash/Mpp/MinTSOScheduler.cpp Show resolved Hide resolved

dbms/src/Flash/Mpp/MinTSOScheduler.cpp Show resolved Hide resolved

dbms/src/Flash/Mpp/MinTSOScheduler.h Outdated Show resolved Hide resolved

dbms/src/Flash/Mpp/MinTSOScheduler.cpp Outdated Show resolved Hide resolved

guo-shaoge added 3 commits September 20, 2023 10:55

Merge branch 'resource_control_part_4' of github.com:guo-shaoge/tifla…

7d7e4e9

…sh into resource_control_part_4

scheduler_entries -> group_entries; mustGetGroupEntry

c47e0a9

Signed-off-by: guo-shaoge <[email protected]>

add metric for group_entry count

d4cd18b

Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge requested a review from windtalker September 20, 2023 03:48

refine metric

bae1018

Signed-off-by: guo-shaoge <[email protected]>

SeaRise approved these changes Sep 20, 2023

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Sep 20, 2023

fill_rate_ms

59c0e70

Signed-off-by: guo-shaoge <[email protected]>

windtalker approved these changes Sep 20, 2023

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 20, 2023

Merge branch 'master' into resource_control_part_4

eadc7c4

ti-chi-bot bot merged commit 939a18d into pingcap:master Sep 20, 2023

guo-shaoge added a commit to guo-shaoge/tiflash that referenced this pull request Sep 25, 2023

Revert "add min tso scheduler for each resource group (pingcap#8072)"

3bbe0c3

This reverts commit 939a18d. Signed-off-by: guo-shaoge <[email protected]>

guo-shaoge mentioned this pull request Sep 25, 2023

Revert "add min tso scheduler for each resource group (#8072)" #8138

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add min tso scheduler for each resource group #8072

add min tso scheduler for each resource group #8072

guo-shaoge commented Sep 8, 2023 •

edited

Loading

guo-shaoge Sep 14, 2023

guo-shaoge commented Sep 14, 2023

guo-shaoge Sep 14, 2023

guo-shaoge Sep 20, 2023

guo-shaoge commented Sep 14, 2023

guo-shaoge commented Sep 14, 2023

guo-shaoge commented Sep 15, 2023

guo-shaoge commented Sep 19, 2023

guo-shaoge commented Sep 19, 2023

SeaRise left a comment

SeaRise Sep 20, 2023

guo-shaoge Sep 20, 2023

windtalker left a comment

ti-chi-bot bot commented Sep 20, 2023

ti-chi-bot bot commented Sep 20, 2023

ti-chi-bot bot commented Sep 20, 2023

add min tso scheduler for each resource group #8072

add min tso scheduler for each resource group #8072

Conversation

guo-shaoge commented Sep 8, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

guo-shaoge Sep 14, 2023

Choose a reason for hiding this comment

guo-shaoge commented Sep 14, 2023

guo-shaoge Sep 14, 2023

Choose a reason for hiding this comment

guo-shaoge Sep 20, 2023

Choose a reason for hiding this comment

guo-shaoge commented Sep 14, 2023

guo-shaoge commented Sep 14, 2023

guo-shaoge commented Sep 15, 2023

guo-shaoge commented Sep 19, 2023

guo-shaoge commented Sep 19, 2023

SeaRise left a comment

Choose a reason for hiding this comment

SeaRise Sep 20, 2023

Choose a reason for hiding this comment

guo-shaoge Sep 20, 2023

Choose a reason for hiding this comment

windtalker left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Sep 20, 2023

ti-chi-bot bot commented Sep 20, 2023

[LGTM Timeline notifier]

ti-chi-bot bot commented Sep 20, 2023

guo-shaoge commented Sep 8, 2023 •

edited

Loading