Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

util/stmtsummary: discard the plan if it is too long and enlarge the tidb_stmt_summary_max_stmt_count value to 3000 #25843

Merged
merged 12 commits into from
Jul 1, 2021
Merged

util/stmtsummary: discard the plan if it is too long and enlarge the tidb_stmt_summary_max_stmt_count value to 3000 #25843

merged 12 commits into from
Jul 1, 2021

Conversation

crazycs520
Copy link
Contributor

Signed-off-by: crazycs [email protected]

What problem does this PR solve?

  1. Currently statements_summary doesn't restrict the plan size, in some extreme scenarios, TIDB might OOM. This PR discards the statements_summary plan if it is too long (encode plan size more than 1MB) to avoid occupies too much memory.

  2. Enlarge tidb_stmt_summary_max_stmt_count default value from 200 to 3000. Since 200 is too small when troubleshooting.

What is changed and how it works?

About Enlarge tidb_stmt_summary_max_stmt_count default value, this doc record the test process. The final conclusion is:
Enlarge the tidb_stmt_summary_max_stmt_count default value to 3000:

  • If the SQL size is 1024, cache 3000 statements only consume 55MB, memory.
  • If the SQL size is 10240, cache 3000 statements only consume 200MB memory.

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:

Check List

Tests

  • Unit test
  • Manual test

Side effects

  • Performance regression
    • Consumes more MEM
  • Breaking backward compatibility

Release note

  • Enlarge the variable tidb_stmt_summary_max_stmt_count default value from 200 to 3000

@crazycs520 crazycs520 added the sig/sql-infra SIG: SQL Infra label Jun 30, 2021
@ti-chi-bot ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jun 30, 2021
@crazycs520
Copy link
Contributor Author

@breeswish @djshow832 PTAL

breezewish
breezewish reviewed Jun 30, 2021
View reviewed changes
util/stmtsummary/statement_summary.go Outdated
if ssElement.samplePlan == discardPlanCauseTooLong {
plan = discardPlanCauseTooLong
} else {
logutil.BgLogger().Error("decode plan in statement summary failed", zap.String("plan", ssElement.samplePlan), zap.String("query", ssElement.sampleSQL), zap.Error(err))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will leak sensitive information even when log redaction is on?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, If the log redaction is on, ssElement.sampleSQL was already been redact, and ssElement.samplePlan since can't been decoded, so it will not leak sensitive information.

@crazycs520
address comment
067ee12 
Signed-off-by: crazycs <[email protected]>
@github-actions github-actions bot added the sig/planner SIG: Planner label Jun 30, 2021
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 1, 2021
@ti-chi-bot
Copy link
Member

@breeswish: Thanks for your review. The bot only counts approvals from reviewers and higher roles in list, but you're still welcome to leave your comments.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@crazycs520 crazycs520 added sig/diagnosis SIG: Diagnosis and removed sig/planner SIG: Planner labels Jul 1, 2021
@github-actions github-actions bot added the sig/planner SIG: Planner label Jul 1, 2021
@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • breeswish
  • djshow832

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jul 1, 2021
@crazycs520
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 57fb71d

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 1, 2021
@ti-chi-bot ti-chi-bot merged commit c4424c2 into pingcap:master Jul 1, 2021
@breezewish
Copy link
Member

/run-cherry-picker

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jul 1, 2021
@crazycs520 @ti-srebot
cherry pick pingcap#25843 to release-4.0
818e969 
Signed-off-by: ti-srebot <[email protected]>
@ti-srebot ti-srebot mentioned this pull request Jul 1, 2021
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #25872

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jul 1, 2021
@crazycs520 @ti-srebot
cherry pick pingcap#25843 to release-5.0
50f3f32 
Signed-off-by: ti-srebot <[email protected]>
@ti-srebot ti-srebot mentioned this pull request Jul 1, 2021
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-5.0 in PR #25873

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jul 1, 2021
@crazycs520 @ti-srebot
cherry pick pingcap#25843 to release-5.1
a3efc29 
Signed-off-by: ti-srebot <[email protected]>
@ti-srebot ti-srebot mentioned this pull request Jul 1, 2021
Merged
@ti-srebot
Copy link
Contributor

cherry pick to release-5.1 in PR #25874

@crazycs520 crazycs520 deleted the discard-too-long-plan branch July 1, 2021 17:27
ti-chi-bot pushed a commit that referenced this pull request Jul 14, 2021
@ti-srebot
util/stmtsummary: discard the plan if it is too long and enlarge the …
67b641d 
…tidb_stmt_summary_max_stmt_count value to 3000 (#25843) (#25872)
ti-chi-bot pushed a commit that referenced this pull request Jul 15, 2021
@ti-srebot
util/stmtsummary: discard the plan if it is too long and enlarge the …
59f7318 
…tidb_stmt_summary_max_stmt_count value to 3000 (#25843) (#25874)
@crazycs520 crazycs520 mentioned this pull request Jul 20, 2021
Closed
@breezewish
Copy link
Member

@crazycs520 Remember to update docs.

ti-chi-bot pushed a commit that referenced this pull request Aug 12, 2021
@ti-srebot
util/stmtsummary: discard the plan if it is too long and enlarge the …
05f97c8 
…tidb_stmt_summary_max_stmt_count value to 3000 (#25843) (#25873)
@crazycs520 crazycs520 mentioned this pull request Aug 17, 2021
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@djshow832 djshow832 djshow832 approved these changes

@breezewish breezewish breezewish approved these changes

Assignees
No one assigned
Labels
component/config needs-cherry-pick-release-5.0 needs-cherry-pick-release-5.1 sig/diagnosis SIG: Diagnosis sig/planner SIG: Planner sig/sql-infra SIG: SQL Infra size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@crazycs520 @ti-chi-bot @breezewish @ti-srebot @djshow832