support sql digest

[lysu]

What problem does this PR solve?

Now, we are hard to identity a group of similar SQLs, but in real product most SQLs execute by program is some similar SQLs that only different in parameter part(yes they maybe not use prepare too).

This PR want to generate a digest for sql group, so later we can do some useful sth base on it.

• online add some hint without modify user's code.
• add some sql into blacklist(e.g. some sql are too slow, but client continue send to tidb, we need protect other vital sql's execution, so add them to blacklist)
• help user do prepare in slient..this is complex and current code is far away from that.. - -

What is changed and how it works?

• use lex to replace intLit, stringLit and so on...
• sha256 result

Check List

Tests

• Unit test

Code changes

• Add exported method

---

This change is 

[morgo]

could we not use MD5 if not for backward compatibility? 😂

Yes, I think it is important for compatibility. Although I will clarify that I think it should be a non-requirement to have matching digests to MySQL. It sounds just too difficult.

[morgo]

Two functions to be aware of:

• STATEMENT_DIGEST(statement) - return the digest of a statement
• STATEMENT_DIGEST_TEXT(statement) - return the normalized sql query

Although they are from 8.0, you might find adding them useful for writing tests.

[kennytm]

Interesting. On MySQL 8.0 the hash function is changed from MD5 to SHA-256, but the output definitely isn't the SHA-256 of the normalized string.

mysql> select statement_digest_text('Select 1');
+-----------------------------------+
| statement_digest_text('Select 1') |
+-----------------------------------+
| SELECT ?                          |
+-----------------------------------+
1 row in set (0.00 sec)

mysql> select statement_digest('Select 1');
+------------------------------------------------------------------+
| statement_digest('Select 1')                                     |
+------------------------------------------------------------------+
| d1b44b0c19af710b5a679907e284acd2ddc285201794bc69a2389d77baedddae |
+------------------------------------------------------------------+
1 row in set (0.00 sec)

>>> hashlib.sha256(b'SELECT ?').hexdigest()
'66cbb3a40d4bbd150b75825ad291a6545399f3098fc1079e4d8b5bb061a6a481'

Apparently MySQL hashes the parsed token stream directly (we could do the same here 😉), meaning we're never going to match the digest as discussed before.

[lysu]

@gregwebs @kennytm thanks, I will take a look~

[morgo]

@kennytm wow, looks like it did change to sha256. The second sentence in these manual pages is different:
https://dev.mysql.com/doc/refman/8.0/en/performance-schema-statement-digests.html
https://dev.mysql.com/doc/refman/5.7/en/performance-schema-statement-digests.html

[alivxxx]

LGTM

[zz-jason]

how about:

d := digesterPool.Get().(*digester)
defer func() {
	d.buffer.Reset()
	digesterPool.Put(d)
}() 

and remove line 48~49?

[lysu]

we should better don't use defer as we can in hot-code path and I have do some refactor PTAL :D
https://medium.com/i0exception/runtime-overhead-of-using-defer-in-go-7140d5c40e32

[lysu]

@zz-jason @XuHuaiyu PTAL PTAL thx~

[lysu]

@zz-jason @XuHuaiyu PTAL~

[lysu]

@lamxTyler @zz-jason @XuHuaiyu PTAL~

[XuHuaiyu]

We may need more comments for this commit, which will make this PR easier to be understood.

[XuHuaiyu]

It seems that isNumLit(-1) will return false. So isPrefixByUnary(-1) return false?

SignedLiteral:
	Literal
	{
		$$ = ast.NewValueExpr($1)
	}
|	'+' NumLiteral
	{
		$$ = &ast.UnaryOperationExpr{Op: opcode.Plus, V: ast.NewValueExpr($2)}
	}
|	'-' NumLiteral
	{
		$$ = &ast.UnaryOperationExpr{Op: opcode.Minus, V: ast.NewValueExpr($2)}
	}

NumLiteral:
	intLit
|	floatLit
|	decLit

[lysu]

@XuHuaiyu No, this PR only use lexer without parser, so alway see token stream like [1] or [-, 1] or [+, 1]

[XuHuaiyu]

We need to add comments for these 2 constants.

[lysu]

@zz-jason @XuHuaiyu PTAL

[lysu]

@zz-jason @XuHuaiyu @eurekaka PTAL

[eurekaka]

LGTM

[zz-jason]

LGTM