*: Add tidbCPU/tikvCPU into system tables #55455

yibin87 · 2024-08-16T02:32:45Z

What problem does this PR solve?

Issue Number: close #55542

Problem Summary:
For on-going slow queries, users naturally want to know some basic execution information to determine possible performance bottlenecks. Although "explain for connection" provide executor-level information, it may be too detailed for some users. Relatively speaking, the process list table provides simpler and more user-friendly information. In many scenarios, Tidb/tikv CPU information can help users quickly determine whether they have encountered CPU resource bottlenecks and further locate whether it is a tidb-side or tikv-side problem.
In addition, these memory system tables can be recorded periodically in the future, providing more information for diagnosing historical problems.

What changed and how does it work?

Several details about the changes:

tidbCPU is collected using topsql mechanism, so it works only when "tidb_enable_top_sql" is enabled.
The PR added extra columns in information_schema.processlists, while not changing the outputs of show processlists command, since the later command shows only brief info.
Use connID + sqlID as the global unique label for each command, sqlID is increased by 1 each time. sqlID will turned to 0 when it exceeds the uint64 MAX, thus there is little possibility that later command with smaller sqlID. Since the possibility is very very small, and affects only the observability not functionality, just ignore.

After the PR, the processlist table schema will add two new columns:

The slow_query table schema changes:

The statements_summary table schema changes:

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2024-08-16T02:32:53Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tiprow · 2024-08-16T02:33:06Z

Hi @yibin87. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

yibin87 · 2024-08-21T02:04:58Z

/cc @wshwsh12 @crazycs520 @mornyx

yibin87 · 2024-08-21T03:05:54Z

/retest

tiprow · 2024-08-21T03:06:26Z

@yibin87: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-08-21T05:50:09Z

Codecov Report

Attention: Patch coverage is 96.42857% with 6 lines in your changes missing coverage. Please review.

Project coverage is 57.3975%. Comparing base (487ef06) to head (dc3a491).
Report is 6 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #55455         +/-   ##
=================================================
- Coverage   72.8514%   57.3975%   -15.4539%     
=================================================
  Files          1598       1745        +147     
  Lines        444454     621258     +176804     
=================================================
+ Hits         323791     356587      +32796     
- Misses       100734     240120     +139386     
- Partials      19929      24551       +4622

Flag	Coverage Δ
integration	`39.6520% <42.6751%> (?)`
tiprow_ft	`?`
unit	`72.0888% <92.2619%> (+0.1181%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9567% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.1774% <ø> (+17.4041%)`	⬆️

yibin87 · 2024-08-21T07:25:50Z

/test unit-test

tiprow · 2024-08-21T07:26:13Z

@yibin87: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

yibin87 · 2024-08-21T09:25:02Z

/test unit-test

tiprow · 2024-08-21T09:25:24Z

@yibin87: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

yibin87 · 2024-08-21T09:41:41Z

/test check-dev2

tiprow · 2024-08-21T09:42:03Z

@yibin87: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test check-dev2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

yibin87 · 2024-08-26T06:03:24Z

/test unit-test

Signed-off-by: yibin <[email protected]>

yibin87 · 2024-09-05T05:40:17Z

LGTM, this PR currently misses a lot of fast-sql's cpu-time, I hope your future data-persistence work can fix this problem.

BTW, the cpu_time in the statements_summary system table can be accurate, after the profile is complete, let the profile collector update the data of the statements_summary.

File a new issut to track the statements_summary improvements: #55869

windtalker

lgtm

yudongusa

LGTM

ti-chi-bot · 2024-09-05T05:58:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: crazycs520, D3Hunter, songrijie, windtalker, wshwsh12, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [D3Hunter,crazycs520,windtalker,wshwsh12]
~~pkg/ddl/OWNERS~~ [D3Hunter]
~~pkg/distsql/OWNERS~~ [windtalker]
~~pkg/infoschema/OWNERS~~ [D3Hunter]
~~pkg/sessionctx/variable/OWNERS~~ [yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

yibin87 · 2024-09-05T06:47:48Z

/test pull-lightning-integration-test

tiprow · 2024-09-05T06:48:09Z

@yibin87: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

yibin87 · 2024-09-05T07:03:51Z

/retest

* origin/master: (33 commits) build(deps): bump github.com/prometheus/common from 0.55.0 to 0.57.0 (pingcap#55762) build(deps): bump golang.org/x/sys from 0.24.0 to 0.25.0 (pingcap#55894) planner: fix incorrect estRows with global index and json column (pingcap#55842) ddl, stat: switch to new struct for add/truncate/drop partition (pingcap#55893) planner: hide instance plan cache eviction log if no plan is evicted (pingcap#55918) expression: support tidb encode key function (pingcap#51678) planner: fix incorrect maintenance of `handleColHelper` for recursive CTE (pingcap#55732) executor: some code refine of hash join v2 (pingcap#55887) infoschema, meta: fix wrong auto id after `rename table` (pingcap#55847) ddl/ingest: set `minCommitTS` when detect remote duplicate keys (pingcap#55588) planner: move index advisor into the kernel (pingcap#55874) ddl, stats: switch to new struct for create/truncate table (pingcap#55811) executor: avoid new small objects in probe stage of hash join v2 (pingcap#55855) *: Add tidbCPU/tikvCPU into system tables (pingcap#55455) ddl: use static contexts in `NewReorgCopContext` (pingcap#55823) executor: fix index out of range bug in hash join v2 (pingcap#55864) executor: record index usage for the clustered primary keys (pingcap#55602) domain: load all non-public tables into infoschema (pingcap#55142) test: fix unstable TestShowViewPriv (pingcap#55868) ttl: add `varbinary` case for `TestSplitTTLScanRangesWithBytes` (pingcap#55863) ...

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Aug 16, 2024

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. sig/planner SIG: Planner labels Aug 16, 2024

ti-chi-bot bot removed do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Aug 21, 2024

yibin87 changed the title ~~[WIPI]executor,util,server: Add tidbCPU/tikvCPU for show processlist command~~ *: Add tidbCPU/tikvCPU for show processlist command Aug 21, 2024

ti-chi-bot bot requested review from crazycs520, mornyx and wshwsh12 August 21, 2024 02:05

yibin87 force-pushed the processlist_enhance branch from 4144941 to 425a142 Compare August 21, 2024 02:15

hawkingrei removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 21, 2024

yibin87 force-pushed the processlist_enhance branch from f113418 to 79c1042 Compare August 21, 2024 04:04

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 21, 2024

yibin87 added 17 commits September 5, 2024 11:48

Little refact

cb94a13

Signed-off-by: yibin <[email protected]>

Little update

78658e8

Signed-off-by: yibin <[email protected]>

Revert useless changes

ee9d5ff

Signed-off-by: yibin <[email protected]>

Fix compilation issues

506401f

Signed-off-by: yibin <[email protected]>

Fix build issue

a72f3b0

Signed-off-by: yibin <[email protected]>

Little update

6045b9a

Signed-off-by: yibin <[email protected]>

Fix build issue

f2c0d33

Signed-off-by: yibin <[email protected]>

update ut file

fff5d40

Signed-off-by: yibin <[email protected]>

Update unit tests

3efd658

Signed-off-by: yibin <[email protected]>

update bazel files after rebase

6c4af90

Signed-off-by: yibin <[email protected]>

update with new unit test

f08f6db

Signed-off-by: yibin <[email protected]>

Copy cpu usages when dist sql detaches

d3149c3

Signed-off-by: yibin <[email protected]>

Address comments to integrate profileprocess into topsql

de63ee5

Signed-off-by: yibin <[email protected]>

Fix ut cases

33294bc

Signed-off-by: yibin <[email protected]>

Resolve conflict

c6650fd

Signed-off-by: yibin <[email protected]>

Resolve conflicts

f87e905

Signed-off-by: yibin <[email protected]>

pkg/util/ppcpuusage/cpuusages.go

dc3a491

Signed-off-by: yibin <[email protected]>

yibin87 force-pushed the processlist_enhance branch from d2e29e0 to dc3a491 Compare September 5, 2024 05:13

yibin87 mentioned this pull request Sep 5, 2024

Make tidb_cpu usages accurate in statements_summary table #55869

Open

windtalker approved these changes Sep 5, 2024

View reviewed changes

yudongusa approved these changes Sep 5, 2024

View reviewed changes

ti-chi-bot bot added the approved label Sep 5, 2024

ti-chi-bot bot merged commit fef43c5 into pingcap:master Sep 5, 2024
25 checks passed

yibin87 mentioned this pull request Sep 12, 2024

*: Change cpu usages related representation format #56044

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: Add tidbCPU/tikvCPU into system tables #55455

*: Add tidbCPU/tikvCPU into system tables #55455

yibin87 commented Aug 16, 2024 •

edited

Loading

ti-chi-bot bot commented Aug 16, 2024

tiprow bot commented Aug 16, 2024

yibin87 commented Aug 21, 2024

yibin87 commented Aug 21, 2024

tiprow bot commented Aug 21, 2024

codecov bot commented Aug 21, 2024 •

edited

Loading

yibin87 commented Aug 21, 2024

tiprow bot commented Aug 21, 2024

yibin87 commented Aug 21, 2024

tiprow bot commented Aug 21, 2024

yibin87 commented Aug 21, 2024

tiprow bot commented Aug 21, 2024

yibin87 commented Aug 26, 2024

yibin87 commented Sep 5, 2024

windtalker left a comment

yudongusa left a comment

ti-chi-bot bot commented Sep 5, 2024

yibin87 commented Sep 5, 2024

tiprow bot commented Sep 5, 2024

yibin87 commented Sep 5, 2024

*: Add tidbCPU/tikvCPU into system tables #55455

*: Add tidbCPU/tikvCPU into system tables #55455

Conversation

yibin87 commented Aug 16, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Aug 16, 2024

tiprow bot commented Aug 16, 2024

yibin87 commented Aug 21, 2024

yibin87 commented Aug 21, 2024

tiprow bot commented Aug 21, 2024

codecov bot commented Aug 21, 2024 • edited Loading

Codecov Report

yibin87 commented Aug 21, 2024

tiprow bot commented Aug 21, 2024

yibin87 commented Aug 21, 2024

tiprow bot commented Aug 21, 2024

yibin87 commented Aug 21, 2024

tiprow bot commented Aug 21, 2024

yibin87 commented Aug 26, 2024

yibin87 commented Sep 5, 2024

windtalker left a comment

Choose a reason for hiding this comment

yudongusa left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Sep 5, 2024

yibin87 commented Sep 5, 2024

tiprow bot commented Sep 5, 2024

yibin87 commented Sep 5, 2024

yibin87 commented Aug 16, 2024 •

edited

Loading

codecov bot commented Aug 21, 2024 •

edited

Loading