expression: constraint propagate for '>' and monotonous function #8640

tiancaiamao · 2018-12-10T14:24:45Z

What problem does this PR solve?

Propagate some thing like:

TO_DAYS(col) < TO_DAYS('2001-01-01') && col > '2009-04-03'  => false

What is changed and how it works?

Add a propagate rule ruleColumnGTConst leverage this PR #7643

Some simple rules:

	col > c1, col > c2, c1 > c2 => col > c1
        col > c1, col < c2, c1 >= c2 => false

A more complicate rule:

	col > c1, f(col) < c2, f is monotonous, f(c1) <= c2 => false

Prove:

        col > c1, f is monotonous => f(col) > f(c1)
        f(col) > f(c1), f(col) < c2, f(c1) >= c2 => false

Check List

Tests

Unit test

Related changes

Ref #7516

This change is

tiancaiamao · 2018-12-11T04:48:44Z

PTAL @eurekaka @zz-jason @winoros

expression/constraint_propagation.go

tiancaiamao · 2018-12-29T06:46:17Z

PTAL @winoros

tiancaiamao · 2018-12-30T05:46:31Z

PTAL @winoros

tiancaiamao · 2019-01-02T03:49:02Z

Tiny update. I remove this rule, it's not useful for partition pruning and adds code complexity

col > c1, col > c2, c1 > c2 => col > c1

PTAL @winoros

winoros

lgtm

tiancaiamao · 2019-01-02T05:25:06Z

PTAL @eurekaka @zz-jason

tiancaiamao · 2019-01-03T05:24:42Z

PTAL @eurekaka @zz-jason

eurekaka · 2019-01-03T05:43:12Z

expression/constraint_propagation.go

+	con1, ok = f1.GetArgs()[1].(*Constant)
+	if !ok {
+		return
+	}


We don't consider const op col?

There are only two rules that really useful to partition pruning:

col < const

col >= const

Why?

create table ... partition by col p1 values less than const1 p2 values less than const2 ...

Let's take a look at p2, the filter expressions are:

col >= const1 and col < const2

The final conditions would be:

col >= const1 and col < const2 and query

So the problem become that whether we can propagate:

col >= const1 and query => false col < const2 and query => false

ruleColumnOPConst means use col op const as rule, perfectly cover the above two.

You'll find the PR can't handle the const op col.
It doesn't matter, we can add another rule that transform
const op col to col op const

Then you'll find the rule order matters.

Not a serious problem, we can control the order by
newConstraintSolver(ruleTransformConstOPColumn, ruleColumnOPConst)

I'll make pruning for select ... where to_days(c) > '2018-01-04' work, and improve some cases later.

I'll make pruning for select ... where to_days(c) > '2018-01-04' work, and improve some cases later.

Thant would be great.

zz-jason · 2019-01-03T03:35:03Z

expression/constant_test.go

@@ -32,12 +32,16 @@ var _ = Suite(&testExpressionSuite{})
 type testExpressionSuite struct{}

 func newColumn(id int) *Column {


how about:
s/newColumn/newInt64Column/
s/newColumnWithType/newColumn/

There are so many newColumn in this file https://github.com/pingcap/tidb/blob/3cf386b5ca2300a0e0cebeae64d58eac9fcbff6c/expression/constant_test.go
If the function signature changes, it bring about ~50 (lines) irrelevant changes to this PR.

And this is _test.go, in this file newColumn indicates newInt64Column

eurekaka

LGTM

eurekaka · 2019-01-04T05:30:45Z

/run-all-tests

codecov-io · 2019-01-04T05:41:44Z

Codecov Report

❗ No coverage uploaded for pull request base (master@e0b30fe). Click here to learn what that means.
The diff coverage is 62.22%.

@@            Coverage Diff            @@
##             master    #8640   +/-   ##
=========================================
  Coverage          ?   67.52%           
=========================================
  Files             ?      363           
  Lines             ?    75073           
  Branches          ?        0           
=========================================
  Hits              ?    50693           
  Misses            ?    19905           
  Partials          ?     4475

Impacted Files	Coverage Δ
expression/constraint_propagation.go	`74.67% <62.22%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0b30fe...c2e6940. Read the comment docs.

…ion (#8640)" This reverts commit 3708a83.

tiancaiamao added status/WIP component/expression labels Dec 10, 2018

tiancaiamao added 3 commits December 11, 2018 10:45

expression: constraint propagate for '>' and monotonous function

6cbf528

Merge branch 'master' into propagate-pruning

f966b75

Merge master

e11f0ed

tiancaiamao force-pushed the propagate-pruning branch from dcb25d5 to e11f0ed Compare December 11, 2018 03:24

tiancaiamao removed the status/WIP label Dec 11, 2018

tiancaiamao added 2 commits December 11, 2018 11:26

gofmt

12d9212

fix CI

3376571

fix CI

9a3659b

winoros reviewed Dec 11, 2018

View reviewed changes

tiancaiamao added 2 commits December 29, 2018 14:38

address comment

b3287fc

Merge branch 'master' into propagate-pruning

0f08c3e

tiancaiamao mentioned this pull request Dec 29, 2018

planner,expression: use constraint propagation in partition pruning #8885

Merged

make the code more general and remove useless part

bdcf799

winoros reviewed Jan 2, 2019

View reviewed changes

winoros added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 2, 2019

zz-jason requested review from eurekaka and zz-jason January 2, 2019 14:03

eurekaka reviewed Jan 3, 2019

View reviewed changes

Merge branch 'master' into propagate-pruning

3cf386b

zz-jason reviewed Jan 4, 2019

View reviewed changes

zz-jason requested a review from alivxxx January 4, 2019 04:58

eurekaka reviewed Jan 4, 2019

View reviewed changes

eurekaka added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 4, 2019

Merge branch 'master' into propagate-pruning

ba71802

eurekaka added type/enhancement The issue or PR belongs to an enhancement. status/all tests passed labels Jan 4, 2019

eurekaka approved these changes Jan 4, 2019

View reviewed changes

Merge branch 'master' into propagate-pruning

c2e6940

tiancaiamao merged commit 3708a83 into pingcap:master Jan 4, 2019

tiancaiamao deleted the propagate-pruning branch January 4, 2019 06:06

zimulala added a commit that referenced this pull request Jan 4, 2019

Revert "expression: constraint propagate for '>' and monotonous funct…

9b30cde

…ion (#8640)" This reverts commit 3708a83.

tiancaiamao mentioned this pull request Feb 7, 2020

*: re-implement partition pruning for better performance #14679

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

expression: constraint propagate for '>' and monotonous function #8640

expression: constraint propagate for '>' and monotonous function #8640

tiancaiamao commented Dec 10, 2018 •

edited

Loading

tiancaiamao commented Dec 11, 2018

tiancaiamao commented Dec 29, 2018

tiancaiamao commented Dec 30, 2018

tiancaiamao commented Jan 2, 2019

winoros left a comment

tiancaiamao commented Jan 2, 2019

tiancaiamao commented Jan 3, 2019

eurekaka Jan 3, 2019

tiancaiamao Jan 4, 2019

eurekaka Jan 4, 2019

zz-jason Jan 3, 2019

tiancaiamao Jan 4, 2019

eurekaka left a comment

eurekaka commented Jan 4, 2019

codecov-io commented Jan 4, 2019 •

edited

Loading

		@@ -32,12 +32,16 @@ var _ = Suite(&testExpressionSuite{})
		type testExpressionSuite struct{}

		func newColumn(id int) *Column {

expression: constraint propagate for '>' and monotonous function #8640

expression: constraint propagate for '>' and monotonous function #8640

Conversation

tiancaiamao commented Dec 10, 2018 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

tiancaiamao commented Dec 11, 2018

tiancaiamao commented Dec 29, 2018

tiancaiamao commented Dec 30, 2018

tiancaiamao commented Jan 2, 2019

winoros left a comment

Choose a reason for hiding this comment

tiancaiamao commented Jan 2, 2019

tiancaiamao commented Jan 3, 2019

eurekaka Jan 3, 2019

Choose a reason for hiding this comment

tiancaiamao Jan 4, 2019

Choose a reason for hiding this comment

eurekaka Jan 4, 2019

Choose a reason for hiding this comment

zz-jason Jan 3, 2019

Choose a reason for hiding this comment

tiancaiamao Jan 4, 2019

Choose a reason for hiding this comment

eurekaka left a comment

Choose a reason for hiding this comment

eurekaka commented Jan 4, 2019

codecov-io commented Jan 4, 2019 • edited Loading

Codecov Report

tiancaiamao commented Dec 10, 2018 •

edited

Loading

codecov-io commented Jan 4, 2019 •

edited

Loading