planner, statistics: build new histogram using range information #7921

winoros · 2018-10-16T15:25:19Z

What problem does this PR solve?

Not a problem. To enhance the histColl held by DataSoure's StatsInfo if we will hold it.

What is changed and how it works?

Cut the histograms' buckets by the given range information.
Currently only numeric column's bucket' bound will be changed during this process.
Other column and index will only remove the bucket that have no intersection with ranges.

Check List

Tests

Unit test

Code changes

Has exported function/method change
Has exported variable/fields change

Side effects

Possible performance regression
Increased code complexity

Related changes

Need to update the documentation
Need to be included in the release note

This change is

planner/core/stats.go

statistics/feedback.go

statistics/histogram.go

statistics/scalar.go

statistics/table.go

zz-jason · 2018-10-23T05:05:57Z

@winoros any update?

winoros · 2018-10-24T12:26:27Z

Updated @lamxTyler

winoros · 2018-10-25T08:59:29Z

Please take a look at Histogram.CalcRangeFraction.... Any better idea to do it?

statistics/histogram.go

winoros · 2018-11-12T07:34:07Z

For index, we need to encode ranges if we want to split them by bucket bound while we need to use non-encode ranges to call GetRowCountByIndexRanges.
So we need to encode it, decode it then encode it again to get the result. It's a little inefficiency. So keep it in original logic.

winoros · 2018-11-12T07:40:44Z

@lamxTyler PTAL

statistics/selectivity.go

statistics/histogram.go

alivxxx · 2018-11-15T09:31:09Z

planner/property/stats_info.go

@@ -24,7 +24,7 @@ type StatsInfo struct {
 	RowCount    float64
 	Cardinality []float64

-	HistColl statistics.HistColl
+	HistColl *statistics.HistColl


So it may be nil now? We should check all the places that use it to avoid nil pointer reference.

In this pr, it's safe. DataSource always holds a histogram.

statistics/histogram.go

planner/core/stats.go

statistics/histogram.go

zz-jason

LGTM

zz-jason · 2019-01-11T05:52:50Z

statistics/histogram.go

@@ -465,10 +466,17 @@ func (hg *Histogram) totalRowCount() float64 {
 	return float64(hg.Buckets[hg.Len()-1].Count + hg.NullCount)
 }

+func (hg *Histogram) notNullCount() float64 {
+	if hg.Len() == 0 {


@winoros OK. Could you simplify totalRowCount() in this PR?

Oh, there is a leaving comment

zz-jason · 2019-01-11T08:02:48Z

@winoros Please merge master and resolve conflicts.

zz-jason · 2019-01-14T02:41:06Z

@winoros CI failed, PTAL

winoros · 2019-01-14T03:28:40Z

@zz-jason It's the thing i mentioned internally. Statistics' feedback has something wrong with UINT range.

…lectivity

winoros · 2019-01-14T08:32:31Z

/run-all-tests

winoros · 2019-01-14T08:33:41Z

@lamxTyler I changed the place where calling the SplitRange for feedback. You can see the subset changes for details.

winoros · 2019-01-14T08:35:21Z

Also, add some code when doing ColumnPrune to make sure that if the table has unsigned pk, the generated extra rowid is also unsigned.

alivxxx

LGTM

winoros added 3 commits October 16, 2018 23:10

planner, statistics: build new histogram using range information

a9f7ade

Merge branch 'master' into selectivity

c2b1708

fix check

14be004

winoros added type/enhancement The issue or PR belongs to an enhancement. sig/planner SIG: Planner component/statistics labels Oct 16, 2018

winoros changed the title ~~Selectivity~~ planner, statistics: build new histogram using range information Oct 16, 2018

add unit-test file.

763f5c1

alivxxx reviewed Oct 18, 2018

View reviewed changes

address comments

5b61c12

winoros added 2 commits October 25, 2018 13:14

Merge branch 'master' into selectivity

caea198

fix strange build failed error

e9df1a3

winoros force-pushed the selectivity branch from 6f352a2 to e9df1a3 Compare October 25, 2018 05:32

fix and add test when there's null, -inf or +inf

37dbb1f

alivxxx reviewed Oct 26, 2018

View reviewed changes

statistics/histogram.go Outdated Show resolved Hide resolved

winoros added 4 commits November 5, 2018 19:51

Merge branch 'master' into selectivity

80819b8

fix bug during merging

c26f85f

address comments for column type

63db3e3

clean code

51a6a5c

zz-jason reviewed Nov 14, 2018

View reviewed changes

address comments

27886bb

winoros force-pushed the selectivity branch from a028508 to 27886bb Compare November 15, 2018 07:07

alivxxx reviewed Nov 15, 2018

View reviewed changes

eurekaka reviewed Nov 15, 2018

View reviewed changes

deal with null

c53f289

alivxxx removed the status/LGT2 Indicates that a PR has LGTM 2. label Jan 7, 2019

address comment

13a6d77

winoros requested review from alivxxx and zz-jason January 8, 2019 05:50

zz-jason previously approved these changes Jan 11, 2019

View reviewed changes

address comment

cefba96

winoros added 2 commits January 11, 2019 16:07

Merge branch 'master' into selectivity

8a7bb77

fix merge error

3a9aa6c

winoros dismissed a stale review via 3a9aa6c January 11, 2019 12:21

Merge branch 'master' into selectivity

1d0a21d

winoros added 3 commits January 14, 2019 16:21

fix feedback error

8006aee

Merge branch 'selectivity' of https://github.com/winoros/tidb into se…

79a211b

…lectivity

remove debug log

b6f57cf

winoros added the status/all tests passed label Jan 14, 2019

alivxxx approved these changes Jan 14, 2019

View reviewed changes

alivxxx added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 14, 2019

alivxxx and others added 2 commits January 14, 2019 19:50

Merge branch 'master' into selectivity

0da8d23

Merge branch 'master' into selectivity

2ee47a1

zz-jason merged commit 82d2726 into pingcap:master Jan 14, 2019

winoros deleted the selectivity branch March 15, 2019 07:41

alivxxx mentioned this pull request Apr 24, 2019

invalid range,range.start should be smaller than range.end #10242

Closed

winoros mentioned this pull request Apr 29, 2019

*: fix bug when unsigned pk meets feedback ranges #10307

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner, statistics: build new histogram using range information #7921

planner, statistics: build new histogram using range information #7921

winoros commented Oct 16, 2018 •

edited by c4pt0r

Loading

zz-jason commented Oct 23, 2018

winoros commented Oct 24, 2018

winoros commented Oct 25, 2018

winoros commented Nov 12, 2018

winoros commented Nov 12, 2018

alivxxx Nov 15, 2018

winoros Jan 8, 2019

zz-jason left a comment

zz-jason Jan 11, 2019

zz-jason commented Jan 11, 2019

zz-jason commented Jan 14, 2019

winoros commented Jan 14, 2019

winoros commented Jan 14, 2019

winoros commented Jan 14, 2019

winoros commented Jan 14, 2019

alivxxx left a comment

planner, statistics: build new histogram using range information #7921

planner, statistics: build new histogram using range information #7921

Conversation

winoros commented Oct 16, 2018 • edited by c4pt0r Loading

What problem does this PR solve?

What is changed and how it works?

Check List

zz-jason commented Oct 23, 2018

winoros commented Oct 24, 2018

winoros commented Oct 25, 2018

winoros commented Nov 12, 2018

winoros commented Nov 12, 2018

alivxxx Nov 15, 2018

Choose a reason for hiding this comment

winoros Jan 8, 2019

Choose a reason for hiding this comment

zz-jason left a comment

Choose a reason for hiding this comment

zz-jason Jan 11, 2019

Choose a reason for hiding this comment

zz-jason commented Jan 11, 2019

zz-jason commented Jan 14, 2019

winoros commented Jan 14, 2019

winoros commented Jan 14, 2019

winoros commented Jan 14, 2019

winoros commented Jan 14, 2019

alivxxx left a comment

Choose a reason for hiding this comment

winoros commented Oct 16, 2018 •

edited by c4pt0r

Loading