parser, infoschema, executor: Add information_schema.keywords #48807

dveeden · 2023-11-22T10:06:08Z

What problem does this PR solve?

Issue Number: close #48801

Problem Summary:

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

The `information_schema.keywords` table was added to make it easier to see what keywords the server has.

ti-chi-bot · 2023-11-22T10:06:12Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

tiprow · 2023-11-22T10:06:27Z

Hi @dveeden. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dveeden · 2023-11-22T10:29:08Z

MySQL:

sql> DESCRIBE information_schema.keywords;
+----------+--------------+------+-----+---------+-------+
| Field    | Type         | Null | Key | Default | Extra |
+----------+--------------+------+-----+---------+-------+
| WORD     | varchar(128) | YES  |     | NULL    |       |
| RESERVED | int          | YES  |     | NULL    |       |
+----------+--------------+------+-----+---------+-------+
2 rows in set (0.0014 sec)

sql> TABLE information_schema.keywords LIMIT 5;
+------------+----------+
| WORD       | RESERVED |
+------------+----------+
| ACCESSIBLE |        1 |
| ACCOUNT    |        0 |
| ACTION     |        0 |
| ACTIVE     |        0 |
| ADD        |        1 |
+------------+----------+
5 rows in set (0.0030 sec)

sql> SELECT RESERVED, COUNT(*) FROM information_schema.KEYWORDS GROUP BY RESERVED;
+----------+----------+
| RESERVED | COUNT(*) |
+----------+----------+
|        1 |      262 |
|        0 |      494 |
+----------+----------+
2 rows in set (0.0020 sec)

sql> SELECT VERSION();
+-----------+
| VERSION() |
+-----------+
| 8.2.0     |
+-----------+
1 row in set (0.0005 sec)

TiDB:

sql> DESCRIBE information_schema.keywords;
+----------+--------------+------+-----+---------+-------+
| Field    | Type         | Null | Key | Default | Extra |
+----------+--------------+------+-----+---------+-------+
| WORD     | varchar(128) | YES  |     | NULL    |       |
| RESERVED | int(11)      | YES  |     | NULL    |       |
+----------+--------------+------+-----+---------+-------+
2 rows in set (0.0012 sec)

sql> TABLE information_schema.keywords LIMIT 5;
+---------+----------+
| WORD    | RESERVED |
+---------+----------+
| ADD     |        1 |
| ALL     |        1 |
| ALTER   |        1 |
| ANALYZE |        1 |
| AND     |        1 |
+---------+----------+
5 rows in set (0.0024 sec)

sql> SELECT RESERVED, COUNT(*) FROM information_schema.KEYWORDS GROUP BY RESERVED;
+----------+----------+
| RESERVED | COUNT(*) |
+----------+----------+
|        1 |      233 |
|        0 |      369 |
+----------+----------+
2 rows in set (0.0012 sec)

sql> SELECT VERSION();
+------------------------------------------+
| VERSION()                                |
+------------------------------------------+
| 8.0.11-TiDB-v7.6.0-alpha-244-gb066ecb37d |
+------------------------------------------+
1 row in set (0.0014 sec)

codecov · 2023-11-22T10:45:00Z

Codecov Report

Merging #48807 (26f4640) into master (a409acb) will increase coverage by 0.9459%.
The diff coverage is 27.7777%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #48807        +/-   ##
================================================
+ Coverage   71.0443%   71.9902%   +0.9459%     
================================================
  Files          1368       1407        +39     
  Lines        402978     424731     +21753     
================================================
+ Hits         286293     305765     +19472     
- Misses        96744      99818      +3074     
+ Partials      19941      19148       -793

Flag	Coverage Δ
integration	`43.6176% <9.0909%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9663% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`47.8520% <ø> (-5.1194%)`	⬇️

dveeden · 2023-11-22T11:20:01Z

/retest

tiprow · 2023-11-22T11:20:25Z

@dveeden: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dveeden · 2023-11-22T11:45:05Z

With MySQL 8.2.0 running locally with this added to the config:

[mysqld]
federated

And setsebool -P mysql_connect_any 1 to configure SELinux.

We can do this:

CREATE TABLE tidb_keywords (
  WORD varchar(128), 
  RESERVED int)
ENGINE=FEDERATED CONNECTION='mysql://[email protected]:4000/information_schema/keywords'

This allows us to compare the two with SQL.

sql> SELECT t.WORD "TiDB Reserved Keywords MySQL doesn't know" FROM tidb_keywords t LEFT JOIN information_schema.keywords m ON t.word=m.word WHERE t.RESERVED=TRUE and m.word IS NULL;
+-------------------------------------------+
| TiDB Reserved Keywords MySQL doesn't know |
+-------------------------------------------+
| CURRENT_ROLE                              |
| ILIKE                                     |
| STATS_EXTENDED                            |
| TiDB_CURRENT_TSO                          |
| TABLESAMPLE                               |
+-------------------------------------------+
5 rows in set (0.2926 sec)

sql> SELECT m.WORD "MySQL Reserved Keywords TiDB doesn't know" FROM information_schema.keywords m LEFT JOIN tidb_keywords t ON t.word=m.word WHERE m.RESERVED=TRUE and t.word IS NULL;
+-------------------------------------------+
| MySQL Reserved Keywords TiDB doesn't know |
+-------------------------------------------+
| ACCESSIBLE                                |
| ASENSITIVE                                |
| BEFORE                                    |
| CONDITION                                 |
| CUBE                                      |
| DEC                                       |
| DETERMINISTIC                             |
| EACH                                      |
| EMPTY                                     |
| GET                                       |
| GROUPING                                  |
| INSENSITIVE                               |
| IO_AFTER_GTIDS                            |
| IO_BEFORE_GTIDS                           |
| JSON_TABLE                                |
| LATERAL                                   |
| LOOP                                      |
| MASTER_BIND                               |
| MASTER_SSL_VERIFY_SERVER_CERT             |
| MODIFIES                                  |
| OPTIMIZER_COSTS                           |
| READS                                     |
| READ_WRITE                                |
| RESIGNAL                                  |
| RETURN                                    |
| SCHEMA                                    |
| SCHEMAS                                   |
| SENSITIVE                                 |
| SIGNAL                                    |
| SPECIFIC                                  |
| UNDO                                      |
+-------------------------------------------+
31 rows in set (0.0204 sec)

sql> SELECT m.WORD,m.RESERVED 'Reserved in MySQL',t.RESERVED 'Reserved in TiDB' FROM information_schema.keywords m INNER JOIN tidb_keywords t ON t.word=m.word WHERE m.RESERVED<>t.RESERVED;
+-----------+-------------------+------------------+
| WORD      | Reserved in MySQL | Reserved in TiDB |
+-----------+-------------------+------------------+
| ARRAY     |                 0 |                1 |
| UNTIL     |                 0 |                1 |
| DECLARE   |                 1 |                0 |
| FUNCTION  |                 1 |                0 |
| PURGE     |                 1 |                0 |
| SEPARATOR |                 1 |                0 |
| SYSTEM    |                 1 |                0 |
+-----------+-------------------+------------------+
7 rows in set (0.7549 sec)

The FEDERATED storage engine has many known issues and isn't recommended for production, but for this case it is a nice tool.

dveeden · 2023-11-22T11:47:25Z

/ok-to-test

pkg/parser/generate_keyword/genkeyword.go

pkg/parser/keywords.go

dveeden · 2023-11-23T13:02:23Z

The new test will fail until #48851 is merged

=== RUN   TestKeywordsSorting
    keywords_test.go:43: CHARACTER should come after CHAR, please update parser.y and re-generate keywords.go
    keywords_test.go:43: CURRENT_USER should come after CURRENT_ROLE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: ELSEIF should come after ELSE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: EXPLAIN should come after EXCEPT, please update parser.y and re-generate keywords.go
    keywords_test.go:43: OUTFILE should come after IS, please update parser.y and re-generate keywords.go
    keywords_test.go:43: IS should come after INSERT, please update parser.y and re-generate keywords.go
    keywords_test.go:43: LIKE should come after ILIKE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: LINES should come after LINEAR, please update parser.y and re-generate keywords.go
    keywords_test.go:43: SQL_SMALL_RESULT should come after SQLEXCEPTION, please update parser.y and re-generate keywords.go
    keywords_test.go:43: TiDB_CURRENT_TSO should come after TABLE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: TABLESAMPLE should come after STORED, please update parser.y and re-generate keywords.go
    keywords_test.go:43: UNIQUE should come after UNION, please update parser.y and re-generate keywords.go
    keywords_test.go:43: UTC_TIMESTAMP should come after UTC_TIME, please update parser.y and re-generate keywords.go
    keywords_test.go:43: VALUES should come after LONG, please update parser.y and re-generate keywords.go
    keywords_test.go:43: VARCHARACTER should come after VARBINARY, please update parser.y and re-generate keywords.go
    keywords_test.go:43: WRITE should come after WINDOW, please update parser.y and re-generate keywords.go
    keywords_test.go:43: ZEROFILL should come after NATURAL, please update parser.y and re-generate keywords.go
    keywords_test.go:43: STATS_SAMPLE_RATE should come after STATS_COL_CHOICE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: STATS_COL_LIST should come after AUTO_ID_CACHE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: BINDING_CACHE should come after BINDINGS, please update parser.y and re-generate keywords.go
    keywords_test.go:43: BOOLEAN should come after BOOL, please update parser.y and re-generate keywords.go
    keywords_test.go:43: COLUMN_FORMAT should come after COLUMNS, please update parser.y and re-generate keywords.go
    keywords_test.go:43: CONFIG should come after COMMENT, please update parser.y and re-generate keywords.go
    keywords_test.go:43: CURRENT should come after CLOSE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: DATETIME should come after DATE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: FOUND should come after FOLLOWING, please update parser.y and re-generate keywords.go
    keywords_test.go:43: IDENTIFIED should come after ERRORS, please update parser.y and re-generate keywords.go
    keywords_test.go:43: LAST_BACKUP should come after LASTVAL, please update parser.y and re-generate keywords.go
    keywords_test.go:43: LOCKED should come after LOCATION, please update parser.y and re-generate keywords.go
    keywords_test.go:43: MAX_MINUTES should come after MAX_CONNECTIONS_PER_HOUR, please update parser.y and re-generate keywords.go
    keywords_test.go:43: MIN_ROWS should come after MINUTE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: NVARCHAR should come after NULLS, please update parser.y and re-generate keywords.go
    keywords_test.go:43: TPCH_10 should come after ON_DUPLICATE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: ON_DUPLICATE should come after ONLINE, please update parser.y and re-generate keywords.go
    keywords_test.go:43: PRE_SPLIT_REGIONS should come after PRECEDING, please update parser.y and re-generate keywords.go
    keywords_test.go:43: RTREE should come after HYPO, please update parser.y and re-generate keywords.go
    keywords_test.go:43: TABLE_CHECKSUM should come after TABLES, please update parser.y and re-generate keywords.go
    keywords_test.go:43: TIMESTAMP should come after TIME, please update parser.y and re-generate keywords.go
    keywords_test.go:43: TYPE should come after TPCC, please update parser.y and re-generate keywords.go
    keywords_test.go:43: YEAR should come after WAIT, please update parser.y and re-generate keywords.go
    keywords_test.go:43: WAIT should come after FAILED_LOGIN_ATTEMPTS, please update parser.y and re-generate keywords.go
--- FAIL: TestKeywordsSorting (0.00s)

dveeden · 2023-12-01T10:23:39Z

/retest

This is expected to fail until #48851 has been merged

ti-chi-bot · 2023-12-01T22:31:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mjonss, okJiang, tangenta

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mjonss,okJiang,tangenta]
~~pkg/infoschema/OWNERS~~ [tangenta]
~~pkg/parser/OWNERS~~ [tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hawkingrei · 2023-12-02T07:07:40Z

/test all

hawkingrei · 2023-12-02T11:17:20Z

/retest

…p#48807)

bb7133 · 2024-06-26T05:06:11Z

/run-cherry-picker

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-06-26T05:06:50Z

In response to a cherrypick label: new pull request created to branch release-7.5: #54219.

Signed-off-by: ti-chi-bot <[email protected]>

#54219) close #48801

ti-chi-bot bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Nov 22, 2023

ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 22, 2023

parser, infoschema, executor: Add information_schema.keywords

b066ecb

dveeden force-pushed the keywords branch from 1f8181c to b066ecb Compare November 22, 2023 10:24

dveeden requested review from tiancaiamao, lance6716 and mjonss November 22, 2023 10:24

dveeden marked this pull request as ready for review November 22, 2023 10:25

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 22, 2023

dveeden added 3 commits November 22, 2023 11:34

Update bazel files

a272fb1

Add test for keywords

b02ac0e

Fix gosec G302

628b7ff

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Nov 22, 2023

tangenta approved these changes Nov 23, 2023

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 23, 2023

mjonss reviewed Nov 23, 2023

View reviewed changes

pkg/parser/generate_keyword/genkeyword.go Show resolved Hide resolved

pkg/parser/generate_keyword/genkeyword.go Show resolved Hide resolved

pkg/parser/keywords.go Outdated Show resolved Hide resolved

dveeden mentioned this pull request Nov 23, 2023

The keywords in parser.y are not sorted properly #48850

Closed

Add more tests

b9d2f49

Merge remote-tracking branch 'upstream/master' into keywords

8702fe5

mjonss approved these changes Dec 1, 2023

View reviewed changes

dveeden added 5 commits December 2, 2023 10:42

Merge remote-tracking branch 'upstream/master' into keywords

8dae134

Restrict what we export in genkeyword

3f37942

Update keywords.go

b3a3170

Add section to keywords to allow testing these correctly

db046a6

Add comments to keywords.go

26f4640

ti-chi-bot bot merged commit a92497b into pingcap:master Dec 2, 2023
16 checks passed

This was referenced Dec 5, 2023

Update keywords (#15443) pingcap/docs#15593

Merged

Update keywords (#15443) pingcap/docs#15594

Merged

dveeden mentioned this pull request Dec 19, 2023

Add docs for the I_S keyword table pingcap/docs#15795

Merged

17 tasks

qiancai mentioned this pull request Dec 20, 2023

Add info about compatibility issues with Connector/J 8.0 with TiDB v7.5 pingcap/docs#15794

Merged

17 tasks

Oreoxmt mentioned this pull request Dec 20, 2023

Add docs for the I_S keyword table pingcap/docs-cn#15834

Merged

17 tasks

dveeden added a commit to dveeden/tidb that referenced this pull request Dec 20, 2023

parser, infoschema, executor: Add information_schema.keywords (pingca…

b3d0986

…p#48807)

dveeden mentioned this pull request Dec 20, 2023

parser: truncate keywords.go when re-creating it #49634

Merged

13 tasks

shawn0915 mentioned this pull request Jan 10, 2024

Add info about compatibility issues with Connector/J 8.0 with TiDB v7.5 pingcap/docs-cn#16121

Merged

17 tasks

bb7133 added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Jun 26, 2024

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jun 26, 2024

This is an automated cherry-pick of pingcap#48807

e8f4668

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jun 26, 2024

parser, infoschema, executor: Add information_schema.keywords (#48807) #54219

Merged

13 tasks

ti-chi-bot mentioned this pull request Jun 26, 2024

parser: truncate keywords.go when re-creating it (#49634) #54221

Merged

13 tasks

breezewish mentioned this pull request Jul 1, 2024

parser: Fix keywords & keywords not checked in CI #54371

Open

13 tasks

bb7133 pushed a commit to ti-chi-bot/tidb that referenced this pull request Jul 2, 2024

This is an automated cherry-pick of pingcap#48807

fb32bea

Signed-off-by: ti-chi-bot <[email protected]>

bb7133 added a commit to bb7133/tidb that referenced this pull request Jul 2, 2024

manually cherry-pick pingcap#48807

aeb49ac

bb7133 mentioned this pull request Jul 2, 2024

parser, infoschema, executor: Add information_schema.keywords (#48807) #54381

Closed

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Jul 4, 2024

parser, infoschema, executor: Add information_schema.keywords (#48807) (

4c7f6a1

#54219) close #48801

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parser, infoschema, executor: Add information_schema.keywords #48807

parser, infoschema, executor: Add information_schema.keywords #48807

dveeden commented Nov 22, 2023

ti-chi-bot bot commented Nov 22, 2023

tiprow bot commented Nov 22, 2023

dveeden commented Nov 22, 2023

codecov bot commented Nov 22, 2023 •

edited

Loading

dveeden commented Nov 22, 2023

tiprow bot commented Nov 22, 2023

dveeden commented Nov 22, 2023 •

edited

Loading

dveeden commented Nov 22, 2023

dveeden commented Nov 23, 2023 •

edited

Loading

dveeden commented Dec 1, 2023 •

edited

Loading

ti-chi-bot bot commented Dec 1, 2023

hawkingrei commented Dec 2, 2023

hawkingrei commented Dec 2, 2023

bb7133 commented Jun 26, 2024

ti-chi-bot commented Jun 26, 2024

parser, infoschema, executor: Add information_schema.keywords #48807

parser, infoschema, executor: Add information_schema.keywords #48807

Conversation

dveeden commented Nov 22, 2023

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Nov 22, 2023

tiprow bot commented Nov 22, 2023

dveeden commented Nov 22, 2023

codecov bot commented Nov 22, 2023 • edited Loading

Codecov Report

dveeden commented Nov 22, 2023

tiprow bot commented Nov 22, 2023

dveeden commented Nov 22, 2023 • edited Loading

dveeden commented Nov 22, 2023

dveeden commented Nov 23, 2023 • edited Loading

dveeden commented Dec 1, 2023 • edited Loading

ti-chi-bot bot commented Dec 1, 2023

hawkingrei commented Dec 2, 2023

hawkingrei commented Dec 2, 2023

bb7133 commented Jun 26, 2024

ti-chi-bot commented Jun 26, 2024

codecov bot commented Nov 22, 2023 •

edited

Loading

dveeden commented Nov 22, 2023 •

edited

Loading

dveeden commented Nov 23, 2023 •

edited

Loading

dveeden commented Dec 1, 2023 •

edited

Loading