statistics: avoid using infoschema when to init stats #54514

hawkingrei · 2024-07-09T04:11:35Z

What problem does this PR solve?

Issue Number: close #55851

Problem Summary:

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

codecov · 2024-07-09T04:58:33Z

Codecov Report

Attention: Patch coverage is 83.08824% with 23 lines in your changes missing coverage. Please review.

Project coverage is 77.5396%. Comparing base (a5e07a2) to head (c420a65).
Report is 72 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #54514        +/-   ##
================================================
+ Coverage   73.4625%   77.5396%   +4.0770%     
================================================
  Files          1605       1649        +44     
  Lines        445096     473612     +28516     
================================================
+ Hits         326979     367237     +40258     
+ Misses        98081      84994     -13087     
- Partials      20036      21381      +1345

Flag	Coverage Δ
integration	`54.3537% <75.7352%> (?)`
unit	`73.7611% <80.8823%> (+1.2181%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9478% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`65.6140% <ø> (+19.5998%)`	⬆️

ti-chi-bot · 2024-08-28T14:05:40Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

hawkingrei · 2024-09-04T07:45:57Z

/retest

hawkingrei · 2024-09-04T07:59:42Z

/retest

hawkingrei · 2024-09-04T10:01:56Z

/retest

Signed-off-by: Weizhen Wang <[email protected]>

hawkingrei · 2024-09-25T07:24:38Z

pkg/statistics/integration_test.go

@@ -508,46 +508,6 @@ func TestIssue44369(t *testing.T) {
 	tk.MustExec("select * from t where a = 10 and bb > 20;")
 }

-// Test the case that after ALTER TABLE happens, the pointer to the column info/index info should be refreshed.
-func TestColAndIdxExistenceMapChangedAfterAlterTable(t *testing.T) {


No table info in the stats. so we don't need to test for it.

hawkingrei · 2024-09-25T07:26:01Z

/retest

hawkingrei · 2024-09-25T07:45:22Z

/retest

Signed-off-by: Weizhen Wang <[email protected]>

hawkingrei · 2024-09-25T08:34:21Z

/retest

hawkingrei · 2024-09-25T09:24:34Z

/retest

tiancaiamao · 2024-09-25T09:30:52Z

pkg/statistics/handle/bootstrap.go

-
-		// Detect the context cancel signal, since it may take a long time for the loop.
-		// TODO: add context to TableInfoByID and remove this code block?
-		if ctx.Err() != nil {


This line is important, why remove it?

Because we don't need to read meta. so we can remove this code.

tiancaiamao · 2024-09-25T09:40:50Z

pkg/statistics/handle/storage/read.go

+	// Now, we cannot init the column info in the ColAndIdxExistenceMap when to disable lite-init-stats.
+	// so we have to get the column info from the domain.
+	is := sctx.GetDomainInfoSchema().(infoschema.InfoSchema)
+	tblInfo, ok := statsHandle.TableInfoByID(is, col.TableID)


What's the difference... it's still calling TableInfoByID?
How could this avoid using infoschema when init stats

In this context, stats are loaded only for the tables that require statistical information, thereby avoiding a full load meta.

pkg/statistics/table.go

Signed-off-by: Weizhen Wang <[email protected]>

hawkingrei · 2024-09-25T16:24:27Z

/retest

hawkingrei · 2024-09-25T16:43:23Z

/retest

ti-chi-bot · 2024-09-29T14:29:54Z

[LGTM Timeline notifier]

Timeline:

2024-09-27 12:50:41.690544279 +0000 UTC m=+13597.110757285: ☑️ agreed by winoros.
2024-09-29 14:29:53.299663543 +0000 UTC m=+192348.719876551: ☑️ agreed by tiancaiamao.

ti-chi-bot · 2024-09-30T01:22:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lance6716, tiancaiamao, winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lance6716,tiancaiamao,winoros]
~~pkg/domain/OWNERS~~ [lance6716]
~~pkg/meta/OWNERS~~ [lance6716]
~~pkg/planner/OWNERS~~ [winoros]
~~pkg/statistics/OWNERS~~ [winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. sig/planner SIG: Planner labels Jul 9, 2024

hawkingrei force-pushed the avoid_use branch from bba0575 to 44d79f6 Compare July 9, 2024 04:13

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 9, 2024

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 9, 2024

hawkingrei force-pushed the avoid_use branch from 8aaecb9 to 250f649 Compare August 13, 2024 04:01

ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 13, 2024

hawkingrei force-pushed the avoid_use branch from 250f649 to a39573d Compare August 23, 2024 03:35

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 28, 2024

hawkingrei force-pushed the avoid_use branch 3 times, most recently from c44c3fa to afa34ce Compare September 4, 2024 06:21

hawkingrei force-pushed the avoid_use branch from 76c1bab to 31185a7 Compare September 4, 2024 08:07

ti-chi-bot bot removed do-not-merge/needs-tests-checked do-not-merge/needs-linked-issue labels Sep 4, 2024

hawkingrei added do-not-merge/needs-linked-issue and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. do-not-merge/needs-linked-issue labels Sep 4, 2024

hawkingrei force-pushed the avoid_use branch from 31185a7 to 5948fde Compare September 5, 2024 03:46

hawkingrei added 4 commits September 25, 2024 14:03

update

bcb3a18

Signed-off-by: Weizhen Wang <[email protected]>

update

789f8b4

Signed-off-by: Weizhen Wang <[email protected]>

update

34f30be

Signed-off-by: Weizhen Wang <[email protected]>

update

2470e5e

Signed-off-by: Weizhen Wang <[email protected]>

hawkingrei force-pushed the avoid_use branch from 76d5f83 to 2470e5e Compare September 25, 2024 06:54

update

d4fe78d

hawkingrei commented Sep 25, 2024

View reviewed changes

update

b922bbb

Signed-off-by: Weizhen Wang <[email protected]>

tiancaiamao reviewed Sep 25, 2024

View reviewed changes

winoros reviewed Sep 25, 2024

View reviewed changes

pkg/statistics/table.go Show resolved Hide resolved

pkg/statistics/table.go Show resolved Hide resolved

hawkingrei added 2 commits September 26, 2024 00:06

update

2c8f49c

Signed-off-by: Weizhen Wang <[email protected]>

update

c420a65

Signed-off-by: Weizhen Wang <[email protected]>

winoros approved these changes Sep 27, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 27, 2024

tiancaiamao approved these changes Sep 29, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 29, 2024

lance6716 approved these changes Sep 30, 2024

View reviewed changes

ti-chi-bot bot added the approved label Sep 30, 2024

ti-chi-bot bot merged commit bb9f4d1 into pingcap:master Sep 30, 2024
25 checks passed

hawkingrei deleted the avoid_use branch September 30, 2024 03:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: avoid using infoschema when to init stats #54514

statistics: avoid using infoschema when to init stats #54514

hawkingrei commented Jul 9, 2024 •

edited

Loading

codecov bot commented Jul 9, 2024 •

edited

Loading

ti-chi-bot bot commented Aug 28, 2024

hawkingrei commented Sep 4, 2024

hawkingrei commented Sep 4, 2024

hawkingrei commented Sep 4, 2024

hawkingrei Sep 25, 2024

hawkingrei commented Sep 25, 2024

hawkingrei commented Sep 25, 2024

hawkingrei commented Sep 25, 2024

hawkingrei commented Sep 25, 2024

tiancaiamao Sep 25, 2024

hawkingrei Sep 26, 2024

tiancaiamao Sep 25, 2024

hawkingrei Sep 25, 2024

hawkingrei commented Sep 25, 2024

hawkingrei commented Sep 25, 2024

ti-chi-bot bot commented Sep 29, 2024

ti-chi-bot bot commented Sep 30, 2024

statistics: avoid using infoschema when to init stats #54514

statistics: avoid using infoschema when to init stats #54514

Conversation

hawkingrei commented Jul 9, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

codecov bot commented Jul 9, 2024 • edited Loading

Codecov Report

ti-chi-bot bot commented Aug 28, 2024

hawkingrei commented Sep 4, 2024

hawkingrei commented Sep 4, 2024

hawkingrei commented Sep 4, 2024

hawkingrei Sep 25, 2024

Choose a reason for hiding this comment

hawkingrei commented Sep 25, 2024

hawkingrei commented Sep 25, 2024

hawkingrei commented Sep 25, 2024

hawkingrei commented Sep 25, 2024

tiancaiamao Sep 25, 2024

Choose a reason for hiding this comment

hawkingrei Sep 26, 2024

Choose a reason for hiding this comment

tiancaiamao Sep 25, 2024

Choose a reason for hiding this comment

hawkingrei Sep 25, 2024

Choose a reason for hiding this comment

hawkingrei commented Sep 25, 2024

hawkingrei commented Sep 25, 2024

ti-chi-bot bot commented Sep 29, 2024

[LGTM Timeline notifier]

ti-chi-bot bot commented Sep 30, 2024

hawkingrei commented Jul 9, 2024 •

edited

Loading

codecov bot commented Jul 9, 2024 •

edited

Loading