*: refactor slow log format and parse slow query log to `SLOW_QUERY` table. #9290

crazycs520 · 2019-02-12T12:28:53Z

What problem does this PR solve?

refactor slow log format to compatible with mysql slow log.
Add INFORMATION_SCHEMA.SLOW_QUERY table, the table content is from parse slow log file.
Add tidb_slow_query_file seesion variable to control which slow log file to be parsed.

What is changed and how it works?

New slow log format is like below, this will be friendly to some utils like pt-query-digest.

# Time: 2019-02-12-19:33:56.571953 +0800
# Txn_start_ts: 406315658548871171
# User: [email protected]
# Conn_ID: 6
# Query_time: 4.895492
# Process_time: 0.161 Request_count: 1 Total_keys: 100001 Processed_keys: 100000
# DB: test
# Is_internal: false
select * from t_slim;

other information like statement digest, plan digest maybe add later.
If you need more information in slow log, please leave you comment.

Query INFORMATION_SCHEMA.SLOW_QUERY table.

mysql> use INFORMATION_SCHEMA;
Database changed
mysql> set @@tidb_slow_query_file="tidb_slow.log";
Query OK, 0 rows affected (0.00 sec)
mysql> select * from `SLOW_QUERY`;
+----------------------------+--------------------+----------------+---------+------------+--------------+-----------+--------------+---------------+------------+----------------+------+-------------+--------------------------------------------------+
| Time                       | Txn_start_ts       | User           | Conn_ID | Query_time | Process_time | Wait_time | Backoff_time | Request_count | Total_keys | Processed_keys | DB   | Is_internal | Query                                            |
+----------------------------+--------------------+----------------+---------+------------+--------------+-----------+--------------+---------------+------------+----------------+------+-------------+--------------------------------------------------+
| 2019-02-12 19:31:06.954777 | 406315605634056194 | root@127.0.0.1 |       1 |  37.142122 |         NULL |      NULL |         NULL |          NULL |       NULL |           NULL | test |           0 | create table t1(y year NOT NULL DEFAULT '2155'); |
| 2019-02-12 19:33:56.571953 | 406315658548871171 | root@127.0.0.1 |       6 |   4.895492 |        0.161 |      NULL |         NULL |             1 |     100001 |         100000 | test |           0 | select * from t_slim;                            |
| 2019-02-12 19:34:33.257245 |                  0 | NULL           |    NULL |    0.51371 |         NULL |      NULL |         NULL |          NULL |       NULL |           NULL | NULL |           1 | analyze table `test`.`t_slim`;                   |
+----------------------------+--------------------+----------------+---------+------------+--------------+-----------+--------------+---------------+------------+----------------+------+-------------+--------------------------------------------------+
3 rows in set (0.00 sec)

mysql> select * from `SLOW_QUERY` where `Is_internal`=false order by `Query_time` desc;
+----------------------------+--------------------+----------------+---------+------------+--------------+-----------+--------------+---------------+------------+----------------+------+-------------+--------------------------------------------------+
| Time                       | Txn_start_ts       | User           | Conn_ID | Query_time | Process_time | Wait_time | Backoff_time | Request_count | Total_keys | Processed_keys | DB   | Is_internal | Query                                            |
+----------------------------+--------------------+----------------+---------+------------+--------------+-----------+--------------+---------------+------------+----------------+------+-------------+--------------------------------------------------+
| 2019-02-12 19:31:06.954777 | 406315605634056194 | root@127.0.0.1 |       1 |  37.142122 |         NULL |      NULL |         NULL |          NULL |       NULL |           NULL | test |           0 | create table t1(y year NOT NULL DEFAULT '2155'); |
| 2019-02-12 19:33:56.571953 | 406315658548871171 | root@127.0.0.1 |       6 |   4.895492 |        0.161 |      NULL |         NULL |             1 |     100001 |         100000 | test |           0 | select * from t_slim;                            |
+----------------------------+--------------------+----------------+---------+------------+--------------+-----------+--------------+---------------+------------+----------------+------+-------------+--------------------------------------------------+
2 rows in set (0.00 sec)

Check List

Tests

Unit test
Integration test

Code changes

Has exported function/method change

Side effects

Related changes

Need to update the documentation

…ll wrong

…g-format

morgo · 2019-02-12T15:56:04Z

Assuming that a statement_digest function is added in pingcap/parser#32, it should be possible to group by digested form :-)

I myself will process the log file in a utility like pt-query-digest, but I can see the use case of presenting in the server (in the absense of pfs for this as in MySQL).

jackysp · 2019-02-13T02:33:38Z

There may be some conflictions between this PR and #9279 .

lysu · 2019-02-13T02:57:51Z

@crazycs520 I have a question, do we keep old slow-log in tidb.log file and put new format into new tidb-slow.log file? or just replace old format with the new one?

new format maybe friendly to pt but it seems not friendly to grep & awk script

morgo · 2019-02-13T03:09:29Z

@crazycs520 I have a question, do we keep old slow-log in tidb.log file and put new format into new tidb-slow.log file? or just replace old format with the new one?

It is a good question. I would rather we don't have two formats. It feels wrong to ask a customer which slow log format they are using when we are suggesting tools to process it.

new format maybe friendly to pt but it seems not friendly to grep & awk script

I think medium term we should offer a pt-query-digest alternative based on https://github.com/Preetam/mysqllog + using our parser for normalization :-) It is quick to build, and will save users from installing perl.

…g-format

executor/adapter.go

util/execdetails/execdetails.go

sessionctx/variable/session.go

infoschema/slow_log.go

infoschema/slow_log_test.go

sessionctx/variable/session.go

crazycs520 · 2019-03-06T02:38:16Z

@zz-jason @zimulala PTAL.

crazycs520 · 2019-03-06T02:53:36Z

/run-all-tests

infoschema/slow_log.go

winkyao

LGTM

sessionctx/variable/session.go

util/execdetails/execdetails.go

infoschema/tables_test.go

infoschema/slow_log.go

Co-Authored-By: crazycs520 <[email protected]>

…g-format

crazycs520 · 2019-03-08T06:49:13Z

@zz-jason PTAL.

infoschema/slow_log.go

util/execdetails/execdetails.go

sessionctx/variable/session.go

sessionctx/variable/session_test.go

util/execdetails/execdetails.go

zimulala · 2019-03-11T03:27:57Z

sessionctx/variable/session.go

+		buf.WriteString(SlowLogPrefixStr + SlowLogIndexNamesStr + SlowLogSpaceMarkStr + indexIDs + "\n")
+	}
+	buf.WriteString(SlowLogPrefixStr + SlowLogIsInternalStr + SlowLogSpaceMarkStr + strconv.FormatBool(s.InRestrictedSQL) + "\n")
+	if len(sql) > 0 {


Under what circumstances will encounter the situation of len(sql) == 0 ?

Normally It won't be len(sql) = 0, but just in case, we should also print ";" when len(sql)=0. Parser need ";" to indicate the end of slow log.

sessionctx/variable/session.go

executor/adapter.go

crazycs520 · 2019-03-11T06:22:35Z

@zz-jason @zimulala PTAL again. Thanks.

zz-jason

LGTM

zz-jason · 2019-03-11T06:34:21Z

/run-all-tests

zimulala · 2019-03-11T06:33:46Z

infoschema/slow_log.go

+	}
+	defer func() {
+		if err = file.Close(); err != nil {
+			log.Error(err)


Please add more information to this log.

zimulala · 2019-03-11T06:42:53Z

sessionctx/variable/session.go

+
+const (
+	// SlowLogPrefixStr is slow log row prefix.
+	SlowLogPrefixStr = "# "


Could we rename it to SlowLogFieldPrefixStr or SlowLogRowPrefixStr?

zimulala · 2019-03-11T06:53:31Z

infoschema/slow_log.go

+	totalKeys    uint64
+	processKeys  uint64
+	db           string
+	indexNames   string


Do we need to update this field name?

…table. (pingcap#9290)

…table. (#9290) (#10060)

crazycs520 added 11 commits January 24, 2019 19:47

slow log init

be5e5a4

use table name, index name instead of id in slow log

31314c4

fix bug, sql should end with ';', otherwise, pt-query-digest parse wi…

e94b4a8

…ll wrong

add slow log table, init

5d60865

parse slow log update

2a6800b

add tidb_slow_query_file session variable

ee25e4c

update test

08a7fb4

fix test

103a354

refine code

a2e68e6

Merge branch 'master' of https://github.com/pingcap/tidb into slow-lo…

00a679f

…g-format

add test for select slow_query table

20342c4

crazycs520 added type/enhancement The issue or PR belongs to an enhancement. type/usability labels Feb 12, 2019

fix ci errcheck

b1deabd

crazycs520 added 2 commits February 27, 2019 10:56

Merge branch 'master' of https://github.com/pingcap/tidb into slow-lo…

2bb3830

…g-format

refine code

9ed2b3b

zz-jason requested review from lysu, winkyao and XuHuaiyu February 27, 2019 03:11

zz-jason reviewed Feb 27, 2019

View reviewed changes

executor/adapter.go Outdated Show resolved Hide resolved

util/execdetails/execdetails.go Outdated Show resolved Hide resolved

lysu reviewed Feb 27, 2019

View reviewed changes

sessionctx/variable/session.go Outdated Show resolved Hide resolved

infoschema/slow_log.go Outdated Show resolved Hide resolved

infoschema/slow_log_test.go Show resolved Hide resolved

address comment and add TODO

bfbb5a1

winkyao reviewed Feb 27, 2019

View reviewed changes

sessionctx/variable/session.go Outdated Show resolved Hide resolved

crazycs520 added 3 commits February 27, 2019 19:36

address comment and add test

c17171a

add comment

2b19b61

fmt code

ab6751e

Merge branch 'master' into slow-log-format

ef4f5cb

winkyao reviewed Mar 6, 2019

View reviewed changes

infoschema/slow_log.go Show resolved Hide resolved

winkyao reviewed Mar 6, 2019

View reviewed changes

Merge branch 'master' into slow-log-format

42800e8

zz-jason reviewed Mar 7, 2019

View reviewed changes

zz-jason and others added 5 commits March 7, 2019 17:34

Update util/execdetails/execdetails.go

982da25

Co-Authored-By: crazycs520 <[email protected]>

Update infoschema/slow_log.go

91a7e46

Co-Authored-By: crazycs520 <[email protected]>

address comment

c9067b9

address comment

5e696c7

Merge branch 'master' of https://github.com/pingcap/tidb into slow-lo…

c5530d6

…g-format

zz-jason reviewed Mar 9, 2019

View reviewed changes

crazycs520 added 2 commits March 11, 2019 11:13

add test for parse time

d8aa948

add test for slow log convert timezone

0b8d096

zimulala reviewed Mar 11, 2019

View reviewed changes

crazycs520 added 3 commits March 11, 2019 12:42

change slow log time type to timestamp

c0993c0

address comment

9161ac7

Merge branch 'master' into slow-log-format

7591f7c

zz-jason approved these changes Mar 11, 2019

View reviewed changes

zz-jason removed the request for review from XuHuaiyu March 11, 2019 06:34

zz-jason merged commit 80110fb into pingcap:master Mar 11, 2019

zimulala reviewed Mar 11, 2019

View reviewed changes

crazycs520 mentioned this pull request Mar 19, 2019

sql: update tidb slow log operation docs pingcap/docs-cn#1182

Merged

crazycs520 added a commit to crazycs520/tidb that referenced this pull request Apr 8, 2019

*: refactor slow log format and parse slow query log to SLOW_QUERY …

3846632

…table. (pingcap#9290)

crazycs520 mentioned this pull request Apr 8, 2019

*: refactor slow log format and parse slow query log to SLOW_QUERY table. (#9290) #10060

Merged

zz-jason pushed a commit that referenced this pull request Apr 9, 2019

*: refactor slow log format and parse slow query log to SLOW_QUERY …

011559c

…table. (#9290) (#10060)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: refactor slow log format and parse slow query log to `SLOW_QUERY` table. #9290

*: refactor slow log format and parse slow query log to `SLOW_QUERY` table. #9290

crazycs520 commented Feb 12, 2019 •

edited

Loading

morgo commented Feb 12, 2019

jackysp commented Feb 13, 2019

lysu commented Feb 13, 2019

morgo commented Feb 13, 2019

crazycs520 commented Mar 6, 2019

crazycs520 commented Mar 6, 2019

winkyao left a comment

crazycs520 commented Mar 8, 2019

zimulala Mar 11, 2019

crazycs520 Mar 11, 2019

crazycs520 commented Mar 11, 2019

zz-jason left a comment

zz-jason commented Mar 11, 2019

zimulala Mar 11, 2019

zimulala Mar 11, 2019

zimulala Mar 11, 2019

*: refactor slow log format and parse slow query log to SLOW_QUERY table. #9290

*: refactor slow log format and parse slow query log to SLOW_QUERY table. #9290

Conversation

crazycs520 commented Feb 12, 2019 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

morgo commented Feb 12, 2019

jackysp commented Feb 13, 2019

lysu commented Feb 13, 2019

morgo commented Feb 13, 2019

crazycs520 commented Mar 6, 2019

crazycs520 commented Mar 6, 2019

winkyao left a comment

Choose a reason for hiding this comment

crazycs520 commented Mar 8, 2019

zimulala Mar 11, 2019

Choose a reason for hiding this comment

crazycs520 Mar 11, 2019

Choose a reason for hiding this comment

crazycs520 commented Mar 11, 2019

zz-jason left a comment

Choose a reason for hiding this comment

zz-jason commented Mar 11, 2019

zimulala Mar 11, 2019

Choose a reason for hiding this comment

zimulala Mar 11, 2019

Choose a reason for hiding this comment

zimulala Mar 11, 2019

Choose a reason for hiding this comment

*: refactor slow log format and parse slow query log to `SLOW_QUERY` table. #9290

*: refactor slow log format and parse slow query log to `SLOW_QUERY` table. #9290

crazycs520 commented Feb 12, 2019 •

edited

Loading