Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

infoschema, util: refresh statement summary table periodically #13680

Merged
merged 8 commits into from
Nov 28, 2019
Merged

infoschema, util: refresh statement summary table periodically #13680

merged 8 commits into from
Nov 28, 2019

Conversation

djshow832
Copy link
Contributor

What problem does this PR solve?

Refresh the table events_statements_summary_by_digest periodically.

What is changed and how it works?

If the current summary exists for over tidb_stmt_summary_refresh_interval seconds, then replace it with a new one.

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)
mysql> select * from events_statements_summary_by_digest limit 1\G
*************************** 1. row ***************************
       SUMMARY_BEGIN_TIME: 2019-11-22 13:00:00
                STMT_TYPE: select
              SCHEMA_NAME: performance_schema
                   DIGEST: 688edff0859079e96540c6161b08dde9ab5f8654bd2755a0dbca27de5e06f93e
              DIGEST_TEXT: select * from events_statements_summary_by_digest
              TABLE_NAMES: performance_schema.events_statements_summary_by_digest
              INDEX_NAMES: NULL
              SAMPLE_USER: [email protected]
               EXEC_COUNT: 1
              SUM_LATENCY: 749473
              MAX_LATENCY: 749473
              MIN_LATENCY: 749473
              AVG_LATENCY: 749473
        AVG_PARSE_LATENCY: 56405
        MAX_PARSE_LATENCY: 56405
      AVG_COMPILE_LATENCY: 444081
      MAX_COMPILE_LATENCY: 444081
             COP_TASK_NUM: 0
     AVG_COP_PROCESS_TIME: 0
     MAX_COP_PROCESS_TIME: 0
  MAX_COP_PROCESS_ADDRESS: NULL
        AVG_COP_WAIT_TIME: 0
        MAX_COP_WAIT_TIME: 0
     MAX_COP_WAIT_ADDRESS: NULL
         AVG_PROCESS_TIME: 0
         MAX_PROCESS_TIME: 0
            AVG_WAIT_TIME: 0
            MAX_WAIT_TIME: 0
         AVG_BACKOFF_TIME: 0
         MAX_BACKOFF_TIME: 0
           AVG_TOTAL_KEYS: 0
           MAX_TOTAL_KEYS: 0
       AVG_PROCESSED_KEYS: 0
       MAX_PROCESSED_KEYS: 0
        AVG_PREWRITE_TIME: 0
        MAX_PREWRITE_TIME: 0
          AVG_COMMIT_TIME: 0
          MAX_COMMIT_TIME: 0
   AVG_GET_COMMIT_TS_TIME: 0
   MAX_GET_COMMIT_TS_TIME: 0
  AVG_COMMIT_BACKOFF_TIME: 0
  MAX_COMMIT_BACKOFF_TIME: 0
    AVG_RESOLVE_LOCK_TIME: 0
    MAX_RESOLVE_LOCK_TIME: 0
AVG_LOCAL_LATCH_WAIT_TIME: 0
MAX_LOCAL_LATCH_WAIT_TIME: 0
           AVG_WRITE_KEYS: 0
           MAX_WRITE_KEYS: 0
           AVG_WRITE_SIZE: 0
           MAX_WRITE_SIZE: 0
     AVG_PREWRITE_REGIONS: 0
     MAX_PREWRITE_REGIONS: 0
            AVG_TXN_RETRY: 0
            MAX_TXN_RETRY: 0
            BACKOFF_TYPES: NULL
                  AVG_MEM: 0
                  MAX_MEM: 0
        AVG_AFFECTED_ROWS: 0
               FIRST_SEEN: 2019-11-22 13:22:53
                LAST_SEEN: 2019-11-22 13:22:53
        QUERY_SAMPLE_TEXT: select * from events_statements_summary_by_digest

Code changes

  • Has exported function/method change

Side effects

  • Possible performance regression
  • Increased code complexity

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Release note

  • Refresh the system table events_statements_summary_by_digest periodically.

@djshow832
Copy link
Contributor Author

/bench

@sre-bot
Copy link
Contributor

sre-bot commented Nov 22, 2019

Benchmark Report

Run Sysbench Performance Test on VMs

@@                               Benchmark Diff                               @@
================================================================================
--- tidb: 86c6e568f120cd046fa118c245d72886978c748a
+++ tidb: 697eb3cf3ecddd036858bc2b2d86a845c3e4322e
tikv: b209a546bb290c60cd975a7b5092b2dfa6464416
pd: 85c7f39a2842ee77a79b014b0cd77fadaa4086b4
================================================================================
oltp_update_non_index:
    * QPS: 4013.57 ± 0.26% (std=8.26) delta: 0.80% (p=0.028)
    * Latency p50: 31.89 ± 0.25% (std=0.07) delta: -0.79%
    * Latency p99: 78.60 ± 0.00% (std=0.00) delta: -1.79%
            
oltp_insert:
    * QPS: 4753.84 ± 0.27% (std=9.83) delta: -0.07% (p=0.684)
    * Latency p50: 26.92 ± 0.27% (std=0.06) delta: 0.08%
    * Latency p99: 48.63 ± 1.19% (std=0.41) delta: 1.47%
            
oltp_read_write:
    * QPS: 15569.47 ± 0.35% (std=32.50) delta: 0.96% (p=0.001)
    * Latency p50: 164.76 ± 0.37% (std=0.36) delta: -0.86%
    * Latency p99: 303.40 ± 3.64% (std=6.75) delta: -2.66%
            
oltp_update_index:
    * QPS: 4284.76 ± 0.04% (std=1.26) delta: 0.06% (p=0.532)
    * Latency p50: 29.87 ± 0.03% (std=0.01) delta: -0.07%
    * Latency p99: 53.38 ± 2.72% (std=1.08) delta: 1.84%
            
oltp_point_select:
    * QPS: 40470.70 ± 0.78% (std=251.07) delta: 0.30% (p=0.660)
    * Latency p50: 3.16 ± 0.87% (std=0.02) delta: -0.24%
    * Latency p99: 9.52 ± 2.23% (std=0.14) delta: -0.89%

@codecov
Copy link

codecov bot commented Nov 22, 2019 •
edited
Loading

Codecov Report

Merging #13680 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #13680   +/-   ##
===========================================
  Coverage   80.1094%   80.1094%           
===========================================
  Files           474        474           
  Lines        116955     116955           
===========================================
  Hits          93692      93692           
  Misses        15866      15866           
  Partials       7397       7397

zimulala
zimulala reviewed Nov 25, 2019
View reviewed changes
sessionctx/variable/tidb_vars.go Outdated
@@ -418,6 +421,7 @@ const (
DefWaitSplitRegionTimeout = 300 // 300s
DefTiDBEnableNoopFuncs = false
DefTiDBAllowRemoveAutoInc = false
DefTiDBStmtSummaryRefreshInterval = 1800
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
DefTiDBStmtSummaryRefreshInterval = 1800
DefTiDBStmtSummaryRefreshInterval = 1800 // 1800s

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed.

AilinKid
AilinKid reviewed Nov 26, 2019
View reviewed changes
sessionctx/variable/varsutil.go Outdated
if v <= 0 {
return value, ErrWrongValueForVar.GenWithStackByArgs(name, value)
}
return value, nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is redundant with line 662.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed.

util/stmtsummary/statement_summary.go Show resolved Hide resolved
util/stmtsummary/statement_summary.go Outdated
@@ -76,6 +86,8 @@ var StmtSummaryByDigestMap = newStmtSummaryByDigestMap()
type stmtSummaryByDigest struct {
// It's rare to read concurrently, so RWMutex is not needed.
sync.Mutex
// Each summary is summarized between [beginTime, beginTime + interval]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Each summary is summarized between [beginTime, beginTime + interval]
// Each summary is summarized between [beginTime, beginTime + interval].

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed.

@djshow832 djshow832 force-pushed the stmt_summary_rolling_table branch 2 times, most recently from 2c48c93 to baf242c Compare November 27, 2019 08:28
@djshow832
Copy link
Contributor Author

/run-all-tests

AilinKid
AilinKid reviewed Nov 27, 2019
View reviewed changes
util/stmtsummary/statement_summary.go

// enabledWrapper encapsulates variables needed to judge whether statement summary is enabled.
enabledWrapper struct {
// sysVars encapsulates system variables needed to control statement summary.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix the comment // only implemented events_statement_summary_by_digest for now. in line 35.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed.

util/stmtsummary/statement_summary.go
return types.MakeDatums(
types.Time{Time: types.FromGoTime(time.Unix(ssbd.beginTime, 0)), Type: mysql.TypeTimestamp},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will time zone take effects?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No.

AilinKid
AilinKid reviewed Nov 28, 2019
View reviewed changes
Copy link
Contributor

@AilinKid AilinKid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@AilinKid AilinKid added status/LGT1 Indicates that a PR has LGTM 1. and removed component/server labels Nov 28, 2019
bb7133
bb7133 reviewed Nov 28, 2019
View reviewed changes
util/stmtsummary/statement_summary.go Outdated
sync.RWMutex
// enabled indicates whether statement summary is enabled in current server.
sessionEnabled string
// setInSession indicates whether statement summary has been set in any session.
globalEnabled string
// XXXRefreshInterval indicates the refresh interval of summaries.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the comment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

XXXRefreshInterval indicates sessionRefreshInterval + globalRefreshInterval + refreshInterval. Is there any better way to express?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed.

util/stmtsummary/statement_summary.go Outdated
return ssMap
}

// AddStatement adds a statement to StmtSummaryByDigestMap.
func (ssMap *stmtSummaryByDigestMap) AddStatement(sei *StmtExecInfo) {
// All times are count in seconds.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// All times are count in seconds.
// All times are counted in seconds.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed.

bb7133
bb7133 approved these changes Nov 28, 2019
View reviewed changes
Copy link
Member

@bb7133 bb7133 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bb7133 bb7133 added status/LGT2 Indicates that a PR has LGTM 2. status/can-merge Indicates a PR has been approved by a committer. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 28, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Nov 28, 2019

Your auto merge job has been accepted, waiting for 13711

@sre-bot
Copy link
Contributor

sre-bot commented Nov 28, 2019

/run-all-tests

@sre-bot sre-bot merged commit 18fbe2d into pingcap:master Nov 28, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Nov 28, 2019

cherry pick to release-3.0 failed

@djshow832
Copy link
Contributor Author

/run-cherry-picker

@sre-bot
Copy link
Contributor

sre-bot commented Dec 20, 2019

cherry pick to release-3.0 failed

djshow832 added a commit to djshow832/tidb that referenced this pull request Dec 20, 2019
@djshow832
infoschema, util: refresh statement summary table periodically (pingc…
8445c73 
…ap#13680)
bb7133 pushed a commit that referenced this pull request Dec 20, 2019
@djshow832 @bb7133
infoschema, util: refresh statement summary table periodically (#13680)…
c6c5fbf 
… (#14161)
XiaTianliang pushed a commit to XiaTianliang/tidb that referenced this pull request Dec 21, 2019
@djshow832 @XiaTianliang
infoschema, util: refresh statement summary table periodically (pingc…
66c7fcd 
…ap#13680)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zimulala zimulala zimulala left review comments

@AilinKid AilinKid AilinKid left review comments

@bb7133 bb7133 bb7133 approved these changes

@crazycs520 crazycs520 Awaiting requested review from crazycs520

Assignees
No one assigned
Labels
status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. type/usability
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@djshow832 @sre-bot @bb7133 @zimulala @AilinKid