*: add HTTP API to generate TiDB metric profile (#18272) #18531
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: ti-srebot <[email protected]>
|
/run-all-tests |
ti-srebot
added
sig/sql-infra
ti-srebot
added
status/LGT2
|
/run-all-tests |
crazycs520
added
status/all-tests-passed
status/can-merge
|
Sorry @crazycs520, you don't have permission to trigger auto merge event on this branch. |
|
/run-all-tests |
|
/rebuild |
|
/run-sqllogic-test-1 |
zz-jason
approved these changes
Jul 28, 2020
ti-srebot
added
status/LGT3
|
/merge |
|
Your auto merge job has been accepted, waiting for:
|
|
/run-all-tests |
|
@ti-srebot merge failed. |
|
/merge |
|
Your auto merge job has been accepted, waiting for:
|
|
/run-all-tests |
|
@ti-srebot merge failed. |
|
/merge |
|
Your auto merge job has been accepted, waiting for:
|
|
/run-all-tests |
|
/run-all-tests |
|
@ti-srebot merge failed. |
|
/merge |
|
/run-all-tests |
|
@ti-srebot merge failed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cherry-pick #18272 to release-4.0
What problem does this PR solve?
TiDB has thousands of metrics, it is hard to know the related metric for users. This PR try to generate a metric profile SVG to describe the total time consumption of related metric in a specified time range.
Below is an example:
The
startandenduse to specify the time range, the time format is RFC3339(2006-01-02T15:04:05Z07:00).The profile.svg look like as below:
From the upper profile SVG, it's easy to see the time consumption ratio of each metric and the relationship between metrics.
Explain
This profile style, color, and format are very similar to the golang pprof profile.
Node
tidb_execute: it was the metric name,tidb_executemeans the consuming time that spends on TiDB execution.20924.55s (30.51%) of 64938.09s (94.68%):20924.55s(30.51%)meanstidb_executeit self consumes20924.55seconds, take up 30.51% of the total time.64938.09s (94.68%)meanstidb_executeitself and all children consume64938.09seconds, take up 94.68% of the total time. So, all the children oftidb_executionconsume64938.09-20924.55=44013.54seconds.Maybe we can add more information about the metric in this node, such as
avg_P99,avg_P80,total_count...Edge
The edge from
tidb_txn_cmdtotidb_txn_cmd.gethas comment with17311.60s, it means17311.60seconds was consumed after this edge.You must found that edges such as the edge from
tidb_txn_cmdtotidb_kv_request, the edge line isdotted, and there is no comment to indicate how much time was consume by its child. This is because the child such astidb_kv_requestmay have multiple parents, I only know the total time consumed bytidb_kv_request, I don't know the time consumed that after the dotted edge.Attention
Currently, the total time was the
tidb_query, so the total time oftidb_queryand its children consumed was 100%.Sometimes, you may found the nodes such as
tidb_kv_request, itself and its children consume time more than thetidb_querytotal time. this is because 1tidb_querymay have many kv requests, and those kv requests may be executed concurrently.For some metric that the total value was less than 0.01% of the total time, the metric will not display in the profile SVG.
Example in different workload
What is changed and how it works?
Each node in the profile has a corresponding metric, such as the metric of
tidb_queryistidb_server_handle_query_duration_seconds.Related changes
pingcap/docs/pingcap/docs-cn:Check List
Tests
Side effects
Release note