Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

planner: add heuristic rules for index selection #26304

Merged
merged 26 commits into from
Aug 4, 2021
Merged

planner: add heuristic rules for index selection #26304

merged 26 commits into from
Aug 4, 2021

Conversation

xuyifangreeneyes
Copy link
Contributor

@xuyifangreeneyes xuyifangreeneyes commented Jul 16, 2021 •
edited
Loading

What problem does this PR solve?

close #26319

What is changed and how it works?

What's Changed & How it Works:

In (*DataSource).DeriveStats, we try heuristic rules to select the path from possible access paths. Once some path matches heuristics, we remove other paths from ds.possibleAccessPaths, let the selected path remain and add a NOTE level warning to indicate the path matches heuristics.

  1. If the query ranges of a unique index are all points and it only needs a single scan, then just use the unique index.
  2. If the query ranges of a unique index are all points but it needs a double scan, we collect it into uniqueIdxsWithDoubleScan. For the one in uniqueIdxsWithDoubleScan with the minimal number of ranges, we record it as uniqueBest.
  3. If the query ranges of an index are not all points but it only needs a single scan, we collect it into singleScanIdxs. For each index in singleScanIdxs, if it is better than some index in uniqueIdxsWithDoubleScan(there is a subset relation between two accessCondsColSet), we call it a refined index. We select the refined index with the minimal number of ranges as refinedBest.
  4. We compare uniqueBest and refinedBest and choose the better one.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note 

add heuristic rules for index selection

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 16, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • qw4990
  • winoros

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jul 16, 2021
@xuyifangreeneyes
Copy link
Contributor Author

/sig planner

@ti-chi-bot ti-chi-bot added the sig/planner SIG: Planner label Jul 16, 2021
@xuyifangreeneyes
Copy link
Contributor Author

/cc @winoros

@ti-chi-bot ti-chi-bot requested a review from winoros July 16, 2021 10:10
@xuyifangreeneyes
Copy link
Contributor Author

/cc @qw4990 @time-and-fate

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 21, 2021
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 28, 2021
@ti-chi-bot ti-chi-bot removed the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 2, 2021
@ti-chi-bot
Copy link
Member

@xuyifangreeneyes: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Aug 2, 2021
@zhouqiang-cl zhouqiang-cl removed the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Aug 4, 2021
qw4990
qw4990 reviewed Aug 4, 2021
View reviewed changes
Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function DataSource.DeriveStats is too huge. It's better to split it into multiple functions, and then put all logic about heuristic rules in one of them?

qw4990
qw4990 reviewed Aug 4, 2021
View reviewed changes
planner/core/stats.go Outdated Show resolved Hide resolved
qw4990
qw4990 reviewed Aug 4, 2021
View reviewed changes
planner/core/stats.go Outdated Show resolved Hide resolved
qw4990
qw4990 reviewed Aug 4, 2021
View reviewed changes
planner/core/integration_test.go
Comment on lines +4159 to +4161
tk.MustExec("drop table if exists t1, t2")
tk.MustExec("create table t1(a int, b int, c int, d int, e int, f int, g int, primary key (a), unique key c_d_e (c, d, e), unique key f (f), unique key f_g (f, g), key g (g))")
tk.MustExec("create table t2(a int, b int, c int, d int, unique index idx_a (a), unique index idx_b_c (b, c), unique index idx_b_c_a_d (b, c, a, d))")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some cases about ClusteredIndex.

qw4990
qw4990 reviewed Aug 4, 2021
View reviewed changes
Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 4, 2021
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 4, 2021
@winoros
Copy link
Member

winoros commented Aug 4, 2021

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 1eddf71

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 4, 2021
@ti-chi-bot
Copy link
Member

@xuyifangreeneyes: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

1 similar comment
@ti-chi-bot
Copy link
Member

@xuyifangreeneyes: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit bc625e9 into pingcap:master Aug 4, 2021
@xuyifangreeneyes xuyifangreeneyes deleted the add-heuristics branch October 30, 2021 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@winoros winoros winoros approved these changes

@qw4990 qw4990 qw4990 approved these changes

@time-and-fate time-and-fate Awaiting requested review from time-and-fate

Assignees
No one assigned
Labels
sig/planner SIG: Planner size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Adding new heuristic rules
5 participants
@xuyifangreeneyes @ti-chi-bot @winoros @qw4990 @zhouqiang-cl