infoschema, util/stmtsummary: enhance statements_summary. #25031

ClSlaid · 2021-06-01T10:19:07Z

What problem does this PR solve?

Issue Number: close #24083

Problem Summary:

What is changed and how it works?

What's Changed:

changed all files in util/stmtsummary
record all evicted information by sum

before:

select * from `INFORMATION_SCHEMA`.`STATEMENTS_SUMMARY`;

output:

BEGIN_TIME	...	DIGEST	SCHEMA_NAME	...
2019-01-01 00:30	...	digest_2	schema_name_2	...
2019-01-01 00:30	...	digest_1	schema_name_1	...
2019-01-01 00:00	...	digest_0	schema_name_0	...

statement_summary table has a fixed size, some statement digests may get kicked by other digests, leading to lose of information.

This pull request will append a special row to the statement_summary, which would sum up those statement digests got evicted from statement_summary, so that all digests' information will be recorded.

Following MySQL's behavior, the row's DIGEST and SCHEMA_NAME match no record in the table, their value is <nil>.

BEGIN_TIME	...	DIGEST	SCHEMA_NAME	...
2019-01-01 00:30	...	digest_2	schema_name_2	...
2019-01-01 00:30	...	digest_1	schema_name_1	...
2019-01-01 00:00	...	digest_0	schema_name_0	...
2019-01-01 00:30	...	`<nil>`	`<nil>`	...

Special rows corresponding to each interval will also be append to statements_summary_history table.

How it Works:

adjust stmtSummaryByDigestEvictedElement in evicted.go, added a class member to record details
add and Mutex to stmtSummaryByDigestEvicted to prevent data racing.

Related changes

Need to cherry-pick to the release branch

Check List

Tests

Under construction

Side effects

Performance regression
- Consumes more CPU
- Consumes more MEM
Breaking backward compatibility

Release note

record all evicted statements information by sum.

Enhance LRU cache, enable users access evicted key-value pair on Put() Signed-off-by: ClSlaid <[email protected]>

Release Note: enhanced LRU Cache some type fixes to cheat CI. Signed-off-by: ClSlaid <[email protected]>

typo fixes. Signed-off-by: ClSlaid <[email protected]>

enhance test on lru's Put() method. Signed-off-by: ClSlaid <[email protected]>

Add a function property in kvcache; Add a method to kvcache; Signed-off-by: ClSlaid <[email protected]>

Used Gofmt Signed-off-by: ClSlaid <[email protected]>

An initial commit, still cannot work properly. Signed-off-by: ClSlaid <[email protected]>

Catch up with PingCAP Signed-off-by: ClSlaid <[email protected]>

now we can see evicted count debug logging expressions not deleted... Signed-off-by: ClSlaid <[email protected]>

STATEMENTS_SUMMARY_EVICTED now able to use Test will be added in tomorrow. Signed-off-by: ClSlaid <[email protected]>

Purge logging code used for debugging. Signed-off-by: ClSlaid <[email protected]>

add test to evicted count, fix bugs in ssMap.other. Signed-off-by: ClSlaid <[email protected]>

Add test to stmtSummaryByDigestMap.ToEvictedCountDatum and stmtSummaryByDigestEvictedElement.ToEvictedCountDatum. Signed-off-by: ClSlaid <[email protected]>

make disable fail-point Signed-off-by: ClSlaid <[email protected]>

� Conflicts: � executor/infoschema_reader.go � infoschema/tables.go

Signed-off-by: ClSlaid <[email protected]>

focus on evicted count and purged unrelated contents. Signed-off-by: ClSlaid <[email protected]>

Signed-off-by: ClSlaid <[email protected]>

improve AddEvicted() from O(mn) to O(n) typo fixs Signed-off-by: ClSlaid <[email protected]>

Signed-off-by: ClSlaid <[email protected]>

ClSlaid · 2021-06-09T14:34:12Z

/run-all-tests

Signed-off-by: ClSlaid <[email protected]>

ClSlaid · 2021-06-10T04:20:26Z

/run-all-tests

ClSlaid · 2021-06-10T04:24:17Z

/run-integration-br-test

ClSlaid · 2021-06-10T04:27:59Z

/run-integration-br-test
/run-check_dev_2

ClSlaid · 2021-06-10T05:36:04Z

/run-integration-br-test

djshow832 · 2021-06-10T06:06:45Z

util/stmtsummary/statement_summary.go

 	rows := make([][]types.Datum, 0, len(values)*historySize)
 	for _, value := range values {
 		records := value.(*stmtSummaryByDigest).toHistoryDatum(historySize, user, isSuper)
 		rows = append(rows, records...)
 	}
+
+	otherDatum := other.toHistoryDatum(historySize)


So you always output other even if there's no other? Does MySQL do the same thing?

NO, you find a bug here...

After examination, such a situation will not happen.
There's always an other that exists, you may refer to the only situation that other contains no record.
You can take a look in stmtSummaryByDigestEvicted.toHistoryDatum().
If there is no record, no history record will be collected from the linked list inside other, so that otherDatum will be an empty list, nothing will be appended to the table.

ClSlaid · 2021-06-10T08:25:17Z

/run-all-tests

ClSlaid · 2021-06-10T11:32:29Z

/run-all-tests

ti-chi-bot · 2021-06-11T02:34:24Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

crazycs520
djshow832

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

djshow832 · 2021-06-11T02:34:35Z

/merge

ti-chi-bot · 2021-06-11T02:34:38Z

This pull request has been accepted and is ready to merge.

Commit hash: ce03218

ClSlaid · 2021-06-11T02:53:08Z

/run-integration-common-test
/run-check_dev_2

ti-chi-bot · 2021-06-11T02:56:44Z

@ClSlaid: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ClSlaid · 2021-06-11T03:07:48Z

/run-integration-common-test

ClSlaid and others added 30 commits April 23, 2021 18:17

util/kvcache: enhance LRU Cache

1a96cbb

Enhance LRU cache, enable users access evicted key-value pair on Put() Signed-off-by: ClSlaid <[email protected]>

Merge branch 'master' into master

0c8920a

Merge branch 'master' into master

25ae806

Util/kvcache: Typo fix

529e346

Release Note: enhanced LRU Cache some type fixes to cheat CI. Signed-off-by: ClSlaid <[email protected]>

util/kvcache/simple_lru.go: typo fix

38fa019

typo fixes. Signed-off-by: ClSlaid <[email protected]>

util/kvcache: fix test

d78ca46

enhance test on lru's Put() method. Signed-off-by: ClSlaid <[email protected]>

Merge branch 'master' into master

1cddd8f

util/kvcache: synced onEvict func -> asynced

33d865f

Add a function property in kvcache; Add a method to kvcache; Signed-off-by: ClSlaid <[email protected]>

Merge branch 'master' of github.com:ClSlaid/tidb

2d75e37

util/kvcache: formatted

05703d8

Used Gofmt Signed-off-by: ClSlaid <[email protected]>

Merge branch 'master' into master

eaf1caa

Merge branch 'master' into master

0163613

Merge branch 'master' into master

0935f75

Statements Summary Evicted Prototype Commit

c0227b6

An initial commit, still cannot work properly. Signed-off-by: ClSlaid <[email protected]>

Merge branch 'master' of github.com:pingcap/tidb

0a168fe

Catch up with PingCAP Signed-off-by: ClSlaid <[email protected]>

Fix STATEMENTS_SUMMARY_EVICTED

b447b86

now we can see evicted count debug logging expressions not deleted... Signed-off-by: ClSlaid <[email protected]>

util/stmtsummary: Fix evicted.go

8e8792b

STATEMENTS_SUMMARY_EVICTED now able to use Test will be added in tomorrow. Signed-off-by: ClSlaid <[email protected]>

Merge branch 'pingcap:master' into master

2a8ed11

Merge branch 'master' into master

12b6a87

util/stmtsummary: Delete debug code

373d344

Purge logging code used for debugging. Signed-off-by: ClSlaid <[email protected]>

util/stmtsummary: add test to EVICTED_COUNT

0f24624

add test to evicted count, fix bugs in ssMap.other. Signed-off-by: ClSlaid <[email protected]>

util/stmtsummary: Add test to evicted count

e2a18b1

Add test to stmtSummaryByDigestMap.ToEvictedCountDatum and stmtSummaryByDigestEvictedElement.ToEvictedCountDatum. Signed-off-by: ClSlaid <[email protected]>

go.sum: disable fail-point

6b7da7e

make disable fail-point Signed-off-by: ClSlaid <[email protected]>

Merge branch 'master' of github.com:pingcap/tidb

ebad974

� Conflicts: � executor/infoschema_reader.go � infoschema/tables.go

fix merge problems

3fcd196

Signed-off-by: ClSlaid <[email protected]>

clean up evicted count

073ff06

focus on evicted count and purged unrelated contents. Signed-off-by: ClSlaid <[email protected]>

typo fix

32e8df9

Signed-off-by: ClSlaid <[email protected]>

performance improve and typo fix

1092e72

improve AddEvicted() from O(mn) to O(n) typo fixs Signed-off-by: ClSlaid <[email protected]>

simplify logic in addEvicted

7e8a8d8

Signed-off-by: ClSlaid <[email protected]>

beautify evicted.go && typo fix

252b43a

Signed-off-by: ClSlaid <[email protected]>

ClSlaid added 3 commits June 9, 2021 22:28

mutex issues :(

ab0cbd8

Signed-off-by: ClSlaid <[email protected]>

Merge branch 'master' into fire

5824872

Merge branch 'fire' of github.com:ClSlaid/tidb into fire

39b1f06

ClSlaid added 2 commits June 10, 2021 12:04

fix firstSeen

337161b

Signed-off-by: ClSlaid <[email protected]>

fmt

39daceb

Signed-off-by: ClSlaid <[email protected]>

djshow832 reviewed Jun 10, 2021

View reviewed changes

Merge branch 'master' into fire

8f0b757

Merge branch 'master' into fire

ce03218

djshow832 approved these changes Jun 11, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 11, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 11, 2021

ti-chi-bot added 2 commits June 11, 2021 10:34

Merge branch 'master' into fire

de53695

Merge branch 'master' into fire

32787d4

Merge branch 'master' into fire

e7b8aff

ti-chi-bot merged commit 487c08a into pingcap:master Jun 11, 2021

ClSlaid mentioned this pull request Aug 12, 2021

Var tidb_stmt_summary_max_stmt_count does not work #27078

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infoschema, util/stmtsummary: enhance statements_summary. #25031

infoschema, util/stmtsummary: enhance statements_summary. #25031

ClSlaid commented Jun 1, 2021 •

edited

Loading

ClSlaid commented Jun 9, 2021

ClSlaid commented Jun 10, 2021

ClSlaid commented Jun 10, 2021

ClSlaid commented Jun 10, 2021

ClSlaid commented Jun 10, 2021

djshow832 Jun 10, 2021

ClSlaid Jun 10, 2021

ClSlaid Jun 10, 2021 •

edited

Loading

ClSlaid commented Jun 10, 2021

ClSlaid commented Jun 10, 2021

ti-chi-bot commented Jun 11, 2021

djshow832 commented Jun 11, 2021

ti-chi-bot commented Jun 11, 2021

ClSlaid commented Jun 11, 2021

ti-chi-bot commented Jun 11, 2021

ClSlaid commented Jun 11, 2021

infoschema, util/stmtsummary: enhance statements_summary. #25031

infoschema, util/stmtsummary: enhance statements_summary. #25031

Conversation

ClSlaid commented Jun 1, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

ClSlaid commented Jun 9, 2021

ClSlaid commented Jun 10, 2021

ClSlaid commented Jun 10, 2021

ClSlaid commented Jun 10, 2021

ClSlaid commented Jun 10, 2021

djshow832 Jun 10, 2021

Choose a reason for hiding this comment

ClSlaid Jun 10, 2021

Choose a reason for hiding this comment

ClSlaid Jun 10, 2021 • edited Loading

Choose a reason for hiding this comment

ClSlaid commented Jun 10, 2021

ClSlaid commented Jun 10, 2021

ti-chi-bot commented Jun 11, 2021

djshow832 commented Jun 11, 2021

ti-chi-bot commented Jun 11, 2021

ClSlaid commented Jun 11, 2021

ti-chi-bot commented Jun 11, 2021

ClSlaid commented Jun 11, 2021

ClSlaid commented Jun 1, 2021 •

edited

Loading

ClSlaid Jun 10, 2021 •

edited

Loading