executor: show operators' memory consumption in results of `EXPLAIN ANALYZE` #11334

qw4990 · 2019-07-19T06:18:54Z

What problem does this PR solve?

Add a new column named memory in the results of EXPLAIN ANALYZE to display memory consumption of operators.

After this PR, you can see:

mysql> explain analyze select * from t order by v limit 5;
+----------------------+-------+------+------------------------------------------------------------+----------------------------------+-----------+
| id                   | count | task | operator info                                              | execution info                   | memory    |
+----------------------+-------+------+------------------------------------------------------------+----------------------------------+-----------+
| TopN_9               | 5.00  | root | test.t.v:asc, offset:0, count:5                            | time:0s, loops:0, rows:0         | 800 Bytes |
| └─TableReader_17     | 5.00  | root | data:TableScan_16                                          | time:261.366µs, loops:2, rows:5  | 131 Bytes |
|   └─TableScan_16     | 5.00  | cop  | table:t, range:[-inf,+inf], keep order:false, stats:pseudo | time:136.023µs, loops:6, rows:5  | N/A         |
+----------------------+-------+------+------------------------------------------------------------+----------------------------------+-----------+

mysql> explain analyze select /*+ TIDB_HJ(t1, t2) */ t1.k from t t1, t t2 where t1.v = t2.v+1;
+--------------------------+-------+------+----------------------------------------------------------------------------+----------------------------------+---------------+
| id                       | count | task | operator info                                                              | execution info                   | memory        |
+--------------------------+-------+------+----------------------------------------------------------------------------+----------------------------------+---------------+
| Projection_8             | 6.25  | root | test.t1.k                                                                  | time:0s, loops:0, rows:0         | N/A             |
| └─HashLeftJoin_9         | 6.25  | root | inner join, inner:Projection_14, equal:[eq(test.t1.v, plus(test.t2.v, 1))] | time:280.29µs, loops:1, rows:0   | 8.61328125 KB |
|   ├─TableReader_13       | 5.00  | root | data:TableScan_12                                                          | time:171.233µs, loops:2, rows:5  | 131 Bytes     |
|   │ └─TableScan_12       | 5.00  | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo                | time:117.687µs, loops:6, rows:5  | N/A             |
|   └─Projection_14        | 5.00  | root | test.t2.v, plus(test.t2.v, 1)                                              | time:0s, loops:0, rows:0         | N/A             |
|     └─TableReader_16     | 5.00  | root | data:TableScan_15                                                          | time:149.443µs, loops:2, rows:5  | 121 Bytes     |
|       └─TableScan_15     | 5.00  | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo                | time:92.487µs, loops:6, rows:5   | N/A             |
+--------------------------+-------+------+----------------------------------------------------------------------------+----------------------------------+---------------+

Pipeliner operators like Selection or Projection have no memory consumption.
Cop-tasks' memory consumption will be added by another PR.

What is changed and how it works?

Main changes:

Don't detach trackers when closing executor otherwise we can't get their memory consumption Information and all trackers will be cleaned by ResetContextOfStmt when the next query comes;
Add a new column named memory in results of EXPLAIN ANALYZE;
Rename some trackers' labels;

Check List

Tests

Unit test

codecov · 2019-07-19T06:50:36Z

Codecov Report

Merging #11334 into master will increase coverage by 0.0279%.
The diff coverage is 100%.

@@               Coverage Diff                @@
##             master     #11334        +/-   ##
================================================
+ Coverage   81.3024%   81.3303%   +0.0279%     
================================================
  Files           423        423                
  Lines         90616      90591        -25     
================================================
+ Hits          73673      73678         +5     
+ Misses        11619      11596        -23     
+ Partials       5324       5317         -7

executor/distsql.go

executor/table_reader.go

planner/core/common_plans.go

zz-jason

LGTM

XuHuaiyu · 2019-07-23T12:41:52Z

executor/distsql.go

@@ -415,14 +417,16 @@ func (e *IndexLookUpExecutor) startIndexWorker(ctx context.Context, kvRanges []k
 		e.dagPB.CollectExecutionSummaries = &collExec
 	}

+	tracker := memory.NewTracker(stringutil.StringerStr("InnerWorker"), e.ctx.GetSessionVars().MemQuotaIndexLookupReader)


s/ InnerWorker/ IndexWorker ?

XuHuaiyu · 2019-07-23T12:42:21Z

executor/distsql.go

@@ -486,7 +488,8 @@ func (e *IndexLookUpExecutor) startTableWorker(ctx context.Context, workCh <-cha
 			keepOrder:      e.keepOrder,
 			handleIdx:      e.handleIdx,
 			isCheckOp:      e.isCheckOp,
-			memTracker:     memory.NewTracker(tableWorkerLabel, -1),
+			memTracker: memory.NewTracker(stringutil.StringerStr(fmt.Sprintf("worker_%v", i)),


s/ worker/ TableWorker ?

lysu · 2019-07-23T11:21:16Z

executor/distsql.go

@@ -486,7 +488,8 @@ func (e *IndexLookUpExecutor) startTableWorker(ctx context.Context, workCh <-cha
 			keepOrder:      e.keepOrder,
 			handleIdx:      e.handleIdx,
 			isCheckOp:      e.isCheckOp,
-			memTracker:     memory.NewTracker(tableWorkerLabel, -1),
+			memTracker: memory.NewTracker(stringutil.StringerStr(fmt.Sprintf("worker_%v", i)),


maybe better not alloc this for normal execution

stringutil.MemoizeStr(func() string { return "worker_" + strconv.Itoa(i) })

What about the other stringutil.StringerStr(xx) ?

that seems cannot be avoided :D

The other stringutil.StringerStr(xx) has no sprintf, so it's OK? 🤔

XuHuaiyu

LGTM

lysu

LGTM

qw4990 · 2019-07-24T02:48:33Z

/run-all-tests

…NALYZE` (pingcap#11334)

sre-bot · 2019-07-24T09:23:32Z

cherry pick to release-2.1 failed

…NALYZE`(#11334) (#11418) All tests passed, auto merged by Bot

sre-bot · 2020-04-07T10:22:14Z

It seems that, not for sure, we failed to cherry-pick this commit to release-2.1. Please comment '/run-cherry-picker' to try to trigger the cherry-picker if we did fail to cherry-pick this commit before. @qw4990 PTAL.

qw4990 added the sig/execution SIG execution label Jul 19, 2019

qw4990 force-pushed the explain-mem2 branch from 249d9c5 to 35dc011 Compare July 19, 2019 06:20

qw4990 requested review from SunRunAway, zz-jason and XuHuaiyu July 19, 2019 06:35

zz-jason reviewed Jul 19, 2019

View reviewed changes

executor/distsql.go Show resolved Hide resolved

executor/table_reader.go Show resolved Hide resolved

planner/core/common_plans.go Outdated Show resolved Hide resolved

qw4990 requested review from lysu and wshwsh12 July 22, 2019 02:28

zz-jason reviewed Jul 23, 2019

View reviewed changes

zz-jason added the status/LGT1 Indicates that a PR has LGTM 1. label Jul 23, 2019

qw4990 force-pushed the explain-mem2 branch from 91ce2ba to cc9a47f Compare July 23, 2019 07:12

XuHuaiyu reviewed Jul 23, 2019

View reviewed changes

lysu reviewed Jul 23, 2019

View reviewed changes

qw4990 force-pushed the explain-mem2 branch from cc9a47f to 1eddd0a Compare July 23, 2019 13:04

qw4990 added 13 commits July 24, 2019 10:35

show memory in explain analyze

ca6457c

format

37d1be2

add UT

ca0beec

update UT

8d5f9c2

fix UT

003b738

format

bcd4613

remove Tracker.Detach

6f2c5a2

fixup

95c8e49

fix UT

ce45ed5

fix CI

61e9f2b

address comments

e7288f4

rename tacker name

13efd92

address comments

f276b08

qw4990 force-pushed the explain-mem2 branch from 1eddd0a to f276b08 Compare July 24, 2019 02:35

XuHuaiyu reviewed Jul 24, 2019

View reviewed changes

lysu approved these changes Jul 24, 2019

View reviewed changes

qw4990 merged commit 1ad073b into pingcap:master Jul 24, 2019

qw4990 added a commit to qw4990/tidb that referenced this pull request Jul 24, 2019

executor: show operators' memory consumption in results of `EXPLAIN A…

edbba86

…NALYZE` (pingcap#11334)

qw4990 mentioned this pull request Jul 24, 2019

executor: show operators' memory consumption in results of EXPLAIN ANALYZE(#11334) #11418

Merged

zz-jason added needs-cherry-pick-2.1 labels Jul 24, 2019

sre-bot pushed a commit that referenced this pull request Jul 25, 2019

executor: show operators' memory consumption in results of `EXPLAIN A…

da97861

…NALYZE`(#11334) (#11418) All tests passed, auto merged by Bot

qw4990 removed the needs-cherry-pick-2.1 label Apr 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: show operators' memory consumption in results of `EXPLAIN ANALYZE` #11334

executor: show operators' memory consumption in results of `EXPLAIN ANALYZE` #11334

qw4990 commented Jul 19, 2019 •

edited

Loading

codecov bot commented Jul 19, 2019 •

edited

Loading

zz-jason left a comment

XuHuaiyu Jul 23, 2019

qw4990 Jul 23, 2019

XuHuaiyu Jul 23, 2019

qw4990 Jul 23, 2019

lysu Jul 23, 2019

qw4990 Jul 23, 2019

XuHuaiyu Jul 23, 2019

lysu Jul 23, 2019

qw4990 Jul 24, 2019

XuHuaiyu left a comment

lysu left a comment

qw4990 commented Jul 24, 2019

sre-bot commented Jul 24, 2019

sre-bot commented Apr 7, 2020

executor: show operators' memory consumption in results of EXPLAIN ANALYZE #11334

executor: show operators' memory consumption in results of EXPLAIN ANALYZE #11334

Conversation

qw4990 commented Jul 19, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

codecov bot commented Jul 19, 2019 • edited Loading

Codecov Report

zz-jason left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XuHuaiyu left a comment

Choose a reason for hiding this comment

lysu left a comment

Choose a reason for hiding this comment

qw4990 commented Jul 24, 2019

sre-bot commented Jul 24, 2019

sre-bot commented Apr 7, 2020

executor: show operators' memory consumption in results of `EXPLAIN ANALYZE` #11334

executor: show operators' memory consumption in results of `EXPLAIN ANALYZE` #11334

qw4990 commented Jul 19, 2019 •

edited

Loading

codecov bot commented Jul 19, 2019 •

edited

Loading