plan, partition: re-implement hash partition pruning to support `in` and `or` and some other functions #18574

imtbkcat · 2020-07-15T05:26:29Z

What problem does this PR solve?

Issue Number: issue

Problem Summary:

Currently hash partition pruning using a naive method to get query range. This method just support find single point value. For condition like in (val1, val2, val3) and expression or expression, there is no way to prune. Moreover, this prune method just support very few partition expression like +-*/ and YEAR, MONTH.

What is changed and how it works?

What's Changed:

The new prune method using ranger to get query range, and for point range, using partition expression to get its partition.
Support prune sql like:
select * from hash_example where key in (1, 2, 3, 4).
select * from hash_example where (key1, key2) in ((1, 2), (3, 4))
select * from hash_example where key = 1 or key = 2

How it Works:

The pseudo code is like this:

prune_hash() {
        for each interval {
    	if (the interval has form "(t1.a, t1.b) = (const1, const2)" ) {
		calculate HASH(part_func(t1.a, t1.b));
		find which partition has records with this hash value and mark it as used;
              } else	{
           		mark all partitions as used;
           		break;
       	}
    }
}

Firstly, exacting columns from partition expression and construct a virtual index.
Next, using DetachCondAndBuildRangeForIndex to resolve the range of this virtual index.
If a range is a point, using expression to eval its value.

Related changes

Need to cherry-pick to the release branch

Check List

Tests

Unit test

Side effects

Breaking backward compatibility

Release note

Support in and or for hash partition pruning

codecov · 2020-07-15T05:45:44Z

Codecov Report

Merging #18574 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #18574   +/-   ##
===========================================
  Coverage   79.5832%   79.5832%           
===========================================
  Files           541        541           
  Lines        146277     146277           
===========================================
  Hits         116412     116412           
  Misses        20538      20538           
  Partials       9327       9327

tiancaiamao

Step1: use query condition to calculate range
Step2: take out range HighValue row, pass it to the partition expression to calculate pos
Step3: pos % hashNum to get the final partition

So it is just use calculate range to handle in, or expression and so on...
Calculate range itself is heavy, I doubt the performance of this version is worse than before?

tiancaiamao · 2020-07-15T10:20:05Z

expression/testdata/partition_pruner_in.json

@@ -6,7 +6,6 @@
      "explain select * from t3 where id = 9 and a = 1",
      "explain select * from t2 where id = 9 and a = -110",
      "explain select * from t1 where id = -17",
-      "explain select * from t2 where id = a and a = b and b = 2",


I suggest comment it out make it not to run, instead of remove this.

tiancaiamao · 2020-07-15T10:25:55Z

expression/testdata/partition_pruner_out.json

+      {
+        "SQL": "explain select * from t6 where a = 7 or a = 6",
+        "Result": [
+          "PartitionUnion_9 60.00 root  ",


The result of this case is not the best?

I will check this

fixed, but i need to make ranger/pruner.go much simpler, there are many repeated there

tiancaiamao · 2020-07-15T10:49:04Z

planner/core/rule_partition_processor.go

+	if len(used) > 0 && used[0] == -1 {
+		return s.makeUnionAllChildren(ds, pi, fullRange(len(pi.Definitions)))
+	}
+	children := make([]LogicalPlan, 0, len(pi.Definitions))


Please avoid repeated code

imtbkcat · 2020-07-15T14:42:15Z

@tiancaiamao I think this method cost nearly as same as the original one. The cost of build range is decide on the number of element in condition. For simple condition like naive equal, they are same and for complex condition, this method avoid full scan.

tiancaiamao · 2020-07-16T15:04:36Z

planner/core/rule_partition_processor.go

+	children := make([]LogicalPlan, 0, len(pi.Definitions))
+	for _, pos := range used {
+		if len(ds.partitionNames) > 0 && !s.findByName(ds.partitionNames, pi.Definitions[pos].Name.L) {
+			// For condition like `from t partition (p1) where a = 5`, but they are conflict, return TableDual directly.


used is an array, if all of the used partitions are not in the selection, return table dual

from t partition (p1) where a = 5 or a = 10 ...

fixed, currently i use makeUnionAllChildren and remove convertToRangeOr to convert used into partitionRangeOR

planner/core/rule_partition_processor.go

lysu · 2020-08-19T11:24:13Z

ping @imtbkcat please resolve conflict if free

planner/core/rule_partition_processor.go

util/ranger/ranger.go

lysu · 2020-08-26T07:16:05Z

LGTM

ti-srebot · 2020-08-26T07:16:07Z

@lysu,Thanks for your review. However, LGTM is restricted to Reviewers or higher roles.See the corresponding SIG page for more information. Related SIGs: planner(slack).

lysu · 2020-08-26T07:16:56Z

@winoros @qw4990 please help look, special how to reduce duplicate between ranger.go and pruner.go

lysu · 2020-08-28T03:20:44Z

/run-all-tests tidb-test=pr/1079

tiancaiamao · 2020-08-28T05:32:48Z

LGTM

ti-srebot · 2020-08-28T05:32:49Z

@tiancaiamao,Thanks for your review. However, LGTM is restricted to Reviewers or higher roles.See the corresponding SIG page for more information. Related SIGs: planner(slack).

winoros · 2020-08-28T06:21:51Z

/run-unit-test

winoros · 2020-08-28T06:34:54Z

/run-unit-test

tiancaiamao · 2020-08-28T06:48:57Z

/run-unit-test

imtbkcat · 2020-08-28T07:25:12Z

/run-unit-test

imtbkcat · 2020-08-28T07:47:58Z

/run-unit-test

…and `or` and some other functions pingcap#18574

imtbkcat added type/enhancement The issue or PR belongs to an enhancement. sig/planner SIG: Planner labels Jul 15, 2020

imtbkcat requested review from lysu and tiancaiamao July 15, 2020 05:26

imtbkcat requested review from a team as code owners July 15, 2020 05:26

imtbkcat requested review from SunRunAway and removed request for a team July 15, 2020 05:26

tiancaiamao reviewed Jul 15, 2020

View reviewed changes

tiancaiamao reviewed Jul 16, 2020

View reviewed changes

imtbkcat force-pushed the new_hash_prune branch from 0caa5d2 to a83e821 Compare July 17, 2020 08:35

lysu reviewed Jul 24, 2020

View reviewed changes

planner/core/rule_partition_processor.go Show resolved Hide resolved

planner/core/rule_partition_processor.go Outdated Show resolved Hide resolved

lysu requested a review from winoros July 24, 2020 08:45

github-actions bot added the component/planner label Aug 4, 2020

AndreMouche removed the component/planner label Aug 11, 2020

lysu added the status/require-change label Aug 19, 2020

imtbkcat force-pushed the new_hash_prune branch from a83e821 to 72df49c Compare August 25, 2020 08:45

lysu reviewed Aug 26, 2020

View reviewed changes

planner/core/rule_partition_processor.go Outdated Show resolved Hide resolved

imtbkcat force-pushed the new_hash_prune branch from a88ba7a to d8d3ccc Compare August 26, 2020 05:05

lysu reviewed Aug 26, 2020

View reviewed changes

util/ranger/ranger.go Outdated Show resolved Hide resolved

lysu requested a review from qw4990 August 26, 2020 07:15

lysu requested a review from eurekaka August 26, 2020 07:17

lysu mentioned this pull request Aug 26, 2020

executor: reduce parameter count for detach ranges methods #19486

Merged

imtbkcat and others added 9 commits August 28, 2020 10:59

address comment

361ee56

fix test and merge master

279fbee

fix lint check

2135ed1

address comment

51786e8

address comment

7b1f902

reuse used to avoid allocate

4b6fd33

clean up useless code

507f5a8

remove more useless

ad04d2a

refine new test case after rebase master

36f01a7

lysu dismissed tiancaiamao’s stale review via 36f01a7 August 28, 2020 03:13

lysu force-pushed the new_hash_prune branch from 642ee21 to 36f01a7 Compare August 28, 2020 03:13

Merge branch 'master' into new_hash_prune

01ea414

lysu requested a review from tiancaiamao August 28, 2020 03:20

Merge branch 'master' into new_hash_prune

cb65320

winoros added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Aug 28, 2020

winoros approved these changes Aug 28, 2020

View reviewed changes

Merge branch 'master' into new_hash_prune

6c05c6f

tiancaiamao merged commit 117b3e4 into pingcap:master Aug 28, 2020

tiancaiamao added a commit to tiancaiamao/tidb that referenced this pull request Dec 29, 2020

plan, partition: re-implement hash partition pruning to support in …

07e2f97

…and `or` and some other functions pingcap#18574

tiancaiamao mentioned this pull request Dec 29, 2020

plan, partition: re-implement hash partition pruning to support in and or and some other functions (#18574) #22061

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

plan, partition: re-implement hash partition pruning to support `in` and `or` and some other functions #18574

plan, partition: re-implement hash partition pruning to support `in` and `or` and some other functions #18574

imtbkcat commented Jul 15, 2020 •

edited

Loading

codecov bot commented Jul 15, 2020 •

edited

Loading

tiancaiamao left a comment

tiancaiamao Jul 15, 2020

tiancaiamao Jul 15, 2020

imtbkcat Jul 15, 2020

imtbkcat Jul 17, 2020

tiancaiamao Jul 15, 2020

imtbkcat commented Jul 15, 2020

tiancaiamao Jul 16, 2020

imtbkcat Jul 17, 2020

lysu commented Aug 19, 2020

lysu commented Aug 26, 2020

ti-srebot commented Aug 26, 2020

lysu commented Aug 26, 2020

lysu commented Aug 28, 2020

tiancaiamao commented Aug 28, 2020

ti-srebot commented Aug 28, 2020

winoros commented Aug 28, 2020

winoros commented Aug 28, 2020

tiancaiamao commented Aug 28, 2020

imtbkcat commented Aug 28, 2020

imtbkcat commented Aug 28, 2020

plan, partition: re-implement hash partition pruning to support in and or and some other functions #18574

plan, partition: re-implement hash partition pruning to support in and or and some other functions #18574

Conversation

imtbkcat commented Jul 15, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

codecov bot commented Jul 15, 2020 • edited Loading

Codecov Report

tiancaiamao left a comment

Choose a reason for hiding this comment

tiancaiamao Jul 15, 2020

Choose a reason for hiding this comment

tiancaiamao Jul 15, 2020

Choose a reason for hiding this comment

imtbkcat Jul 15, 2020

Choose a reason for hiding this comment

imtbkcat Jul 17, 2020

Choose a reason for hiding this comment

tiancaiamao Jul 15, 2020

Choose a reason for hiding this comment

imtbkcat commented Jul 15, 2020

tiancaiamao Jul 16, 2020

Choose a reason for hiding this comment

imtbkcat Jul 17, 2020

Choose a reason for hiding this comment

lysu commented Aug 19, 2020

lysu commented Aug 26, 2020

ti-srebot commented Aug 26, 2020

lysu commented Aug 26, 2020

lysu commented Aug 28, 2020

tiancaiamao commented Aug 28, 2020

ti-srebot commented Aug 28, 2020

winoros commented Aug 28, 2020

winoros commented Aug 28, 2020

tiancaiamao commented Aug 28, 2020

imtbkcat commented Aug 28, 2020

imtbkcat commented Aug 28, 2020

plan, partition: re-implement hash partition pruning to support `in` and `or` and some other functions #18574

plan, partition: re-implement hash partition pruning to support `in` and `or` and some other functions #18574

imtbkcat commented Jul 15, 2020 •

edited

Loading

codecov bot commented Jul 15, 2020 •

edited

Loading