statistics: support tracking column cmsketch #35135

Yisaer · 2022-06-02T09:00:41Z

Signed-off-by: yisaer [email protected]

What problem does this PR solve?

Issue Number: ref #34052

Problem Summary:

What is changed and how it works?

support tracking column cmsketch memory in statsCache

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

statscache test report

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: yisaer <[email protected]>

ti-chi-bot · 2022-06-02T09:00:43Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

chrysan
xuyifangreeneyes

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

sre-bot · 2022-06-02T09:01:10Z

Please follow PR Title Format:

pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

*: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

Yisaer · 2022-06-02T09:02:12Z

/run-check_title

Signed-off-by: yisaer <[email protected]>

sre-bot · 2022-06-06T06:25:20Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/ed536b2842f706994d9158858ed186d8cb407892

Signed-off-by: yisaer <[email protected]> make loaded atomic Signed-off-by: yisaer <[email protected]>

Signed-off-by: yisaer <[email protected]>

Signed-off-by: yisaer <[email protected]> add size metrics Signed-off-by: yisaer <[email protected]> Revert "add size metrics" This reverts commit 05b06a3f4f3ad98ae3f6ca389db2fc8ccc193dc4.

Signed-off-by: yisaer <[email protected]>

Yisaer · 2022-06-13T09:37:24Z

/run-unit-test

xuyifangreeneyes · 2022-06-14T03:11:48Z

statistics/histogram.go

+func (c *Column) DropEvicted() {
+	if c.StatsVer < Version2 {
+		c.CMSketch = nil
+		c.Loaded = false


The comment says "Loaded means if the histogram, the topn and the cm sketch are loaded fully". Before the pr, we assume either all three are loaded or none of the three are loaded. Now we break the assumption. It may bring some problems. For example, (*Column).IsLoaded returns c.Loaded || c.notNullCount() > 0. It seems that we assume c.Loaded is always equal to c.notNullCount() > 0 and will remove c.notNullCount() > 0 in the future. Now if the cm sketch is evicted in ver1 case, c.Loaded is false while c.notNullCount() > 0 is true. cc @time-and-fate

We will revise loaded in next pr

@xuyifangreeneyes is correct.
There're mainly two usages of IsLoaded:

help to check if we can use the non-pseudo estimation logic for this Column.

help to check if stats in this Column needs loading.

And of course, they should be essentially the same thing before this PR.
It seems that they become different in this PR, so we may need to introduce a new field or change IsLoaded to represent more states to handle this.

the revised loaded updated in #35361

Signed-off-by: yisaer <[email protected]>

Yisaer · 2022-06-17T05:54:27Z

/merge

ti-chi-bot · 2022-06-17T05:54:30Z

This pull request has been accepted and is ready to merge.

Commit hash: 532e2df

Yisaer · 2022-06-17T06:16:17Z

/run-unit-test

Yisaer · 2022-06-17T06:16:44Z

/run-build

Yisaer · 2022-06-17T06:16:54Z

/run-mysql-test

Yisaer · 2022-06-17T06:17:10Z

/run-mysql-test

Signed-off-by: yisaer <[email protected]>

…pport_column_cmsketch

Yisaer · 2022-06-17T06:40:07Z

/merge

ti-chi-bot · 2022-06-17T06:40:10Z

This pull request has been accepted and is ready to merge.

Commit hash: 1b8a804

Yisaer · 2022-06-17T07:06:57Z

/run-all-tests

Yisaer · 2022-06-17T07:36:55Z

/run-unit-test

Signed-off-by: yisaer <[email protected]>

Yisaer · 2022-06-17T08:42:51Z

/merge

ti-chi-bot · 2022-06-17T08:42:54Z

This pull request has been accepted and is ready to merge.

Commit hash: ff4158d

sre-bot · 2022-06-17T09:51:59Z

TiDB MergeCI notify

✅ Well Done! New fixed [2] after this pr merged.

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/integration-common-test	✅ all 11 tests passed	25 min	Fixed
idc-jenkins-ci/integration-cdc-test	✅ all 34 tests passed	24 min	Fixed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	13 min	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	13 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 12 tests passed	9 min 16 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	8 min 56 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	8 min 54 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	3 min 48 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	3 min 47 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

support tracking column cmsketch

891b61f

Signed-off-by: yisaer <[email protected]>

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 2, 2022

Yisaer changed the title ~~support tracking column cmsketch~~ statistics: support tracking column cmsketch Jun 2, 2022

fix test

12d4941

Signed-off-by: yisaer <[email protected]>

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 6, 2022

Yisaer closed this Jun 6, 2022

Yisaer reopened this Jun 6, 2022

make loaded atomic

be6d2ab

Signed-off-by: yisaer <[email protected]> make loaded atomic Signed-off-by: yisaer <[email protected]>

Yisaer force-pushed the support_column_cmsketch branch from f29e571 to be6d2ab Compare June 6, 2022 07:45

Yisaer added 2 commits June 7, 2022 13:53

fix

ccaf3a0

Signed-off-by: yisaer <[email protected]>

allow column init cmsketch

7cefc59

Signed-off-by: yisaer <[email protected]> add size metrics Signed-off-by: yisaer <[email protected]> Revert "add size metrics" This reverts commit 05b06a3f4f3ad98ae3f6ca389db2fc8ccc193dc4.

Yisaer force-pushed the support_column_cmsketch branch from 05b06a3 to 7cefc59 Compare June 7, 2022 11:02

Yisaer added 4 commits June 9, 2022 12:39

add testcase

f448110

Signed-off-by: yisaer <[email protected]>

remove useless code

2030048

Signed-off-by: yisaer <[email protected]>

fix test

1651163

Signed-off-by: yisaer <[email protected]>

Merge branch 'master' into support_column_cmsketch

c69e9af

Yisaer marked this pull request as ready for review June 9, 2022 05:18

ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 9, 2022

Yisaer requested review from xuyifangreeneyes and chrysan June 9, 2022 05:19

Merge branch 'master' into support_column_cmsketch

baf94f8

xuyifangreeneyes reviewed Jun 14, 2022

View reviewed changes

address the comment

532e2df

Signed-off-by: yisaer <[email protected]>

chrysan approved these changes Jun 17, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 17, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 17, 2022

Merge branch 'master' into support_column_cmsketch

ff9a72b

Yisaer added 2 commits June 17, 2022 14:39

fix fmt

9f39aa6

Signed-off-by: yisaer <[email protected]>

Merge remote-tracking branch 'origin/support_column_cmsketch' into su…

1b8a804

…pport_column_cmsketch

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Jun 17, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 17, 2022

fix test

7bf2426

Signed-off-by: yisaer <[email protected]>

Yisaer force-pushed the support_column_cmsketch branch from c1b7af9 to 7bf2426 Compare June 17, 2022 08:42

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Jun 17, 2022

Merge branch 'master' into support_column_cmsketch

ff4158d

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 17, 2022

Merge branch 'master' into support_column_cmsketch

ed536b2

ti-chi-bot merged commit 9652651 into pingcap:master Jun 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: support tracking column cmsketch #35135

statistics: support tracking column cmsketch #35135

Yisaer commented Jun 2, 2022 •

edited

Loading

ti-chi-bot commented Jun 2, 2022 •

edited

Loading

sre-bot commented Jun 2, 2022

Yisaer commented Jun 2, 2022

sre-bot commented Jun 6, 2022 •

edited

Loading

Yisaer commented Jun 13, 2022

xuyifangreeneyes Jun 14, 2022

Yisaer Jun 14, 2022 •

edited

Loading

time-and-fate Jun 14, 2022 •

edited

Loading

Yisaer Jun 14, 2022

Yisaer commented Jun 17, 2022

ti-chi-bot commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

ti-chi-bot commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

ti-chi-bot commented Jun 17, 2022

sre-bot commented Jun 17, 2022

statistics: support tracking column cmsketch #35135

statistics: support tracking column cmsketch #35135

Conversation

Yisaer commented Jun 2, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jun 2, 2022 • edited Loading

sre-bot commented Jun 2, 2022

Yisaer commented Jun 2, 2022

sre-bot commented Jun 6, 2022 • edited Loading

Yisaer commented Jun 13, 2022

xuyifangreeneyes Jun 14, 2022

Choose a reason for hiding this comment

Yisaer Jun 14, 2022 • edited Loading

Choose a reason for hiding this comment

time-and-fate Jun 14, 2022 • edited Loading

Choose a reason for hiding this comment

Yisaer Jun 14, 2022

Choose a reason for hiding this comment

Yisaer commented Jun 17, 2022

ti-chi-bot commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

ti-chi-bot commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

Yisaer commented Jun 17, 2022

ti-chi-bot commented Jun 17, 2022

sre-bot commented Jun 17, 2022

TiDB MergeCI notify

Yisaer commented Jun 2, 2022 •

edited

Loading

ti-chi-bot commented Jun 2, 2022 •

edited

Loading

sre-bot commented Jun 6, 2022 •

edited

Loading

Yisaer Jun 14, 2022 •

edited

Loading

time-and-fate Jun 14, 2022 •

edited

Loading