infoschema: adapt TableByName for query of memory table #54918

tangenta · 2024-07-25T14:37:23Z

What problem does this PR solve?

Issue Number: ref #50305

Problem Summary:

Read memory tables from information_schema is slow when tidb_schema_cache_size is non-zero.

What changed and how does it work?

Add TableSchemaSelector to help return the table/schema names specified in predicate.
Adapt TableByName for memory table information_schema.tables.
Adapt FindTableByPartitionID for information_schema.partitions.

Check List

Tests

Unit test
Integration test

Manual test (add detailed scripts or steps below)
Before this PR:

-- first load
mysql> select * from  information_schema.tables where table_schema = 'test' and table_name = 't6613';
+---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | ENGINE | VERSION | ROW_FORMAT | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH | MAX_DATA_LENGTH | INDEX_LENGTH | DATA_FREE | AUTO_INCREMENT | CREATE_TIME         | UPDATE_TIME | CHECK_TIME | TABLE_COLLATION | CHECKSUM | CREATE_OPTIONS | TABLE_COMMENT | TIDB_TABLE_ID | TIDB_ROW_ID_SHARDING_INFO | TIDB_PK_TYPE | TIDB_PLACEMENT_POLICY_NAME |
+---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
| def           | test         | t6613      | BASE TABLE | InnoDB |      10 | Compact    |          2 |              8 |          16 |               0 |           16 |         0 |              5 | 2024-07-25 21:10:37 | NULL        | NULL       | utf8mb4_bin     |     NULL |                |               |          8063 | NOT_SHARDED               | NONCLUSTERED | NULL                       |
+---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
1 row in set (22.68 sec)


-- subsequent load
mysql> select * from  information_schema.tables where table_schema = 'test' and table_name = 't6613';
+---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | ENGINE | VERSION | ROW_FORMAT | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH | MAX_DATA_LENGTH | INDEX_LENGTH | DATA_FREE | AUTO_INCREMENT | CREATE_TIME         | UPDATE_TIME | CHECK_TIME | TABLE_COLLATION | CHECKSUM | CREATE_OPTIONS | TABLE_COMMENT | TIDB_TABLE_ID | TIDB_ROW_ID_SHARDING_INFO | TIDB_PK_TYPE | TIDB_PLACEMENT_POLICY_NAME |
+---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
| def           | test         | t6613      | BASE TABLE | InnoDB |      10 | Compact    |          2 |              8 |          16 |               0 |           16 |         0 |              5 | 2024-07-25 21:10:37 | NULL        | NULL       | utf8mb4_bin     |     NULL |                |               |          8063 | NOT_SHARDED               | NONCLUSTERED | NULL                       |
+---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
1 row in set (2.51 sec)

After this PR:

mysql> select * from  information_schema.tables where table_schema = 'test' and table_name = 't6613';
+---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
| TABLE_CATALOG | TABLE_SCHEMA | TABLE_NAME | TABLE_TYPE | ENGINE | VERSION | ROW_FORMAT | TABLE_ROWS | AVG_ROW_LENGTH | DATA_LENGTH | MAX_DATA_LENGTH | INDEX_LENGTH | DATA_FREE | AUTO_INCREMENT | CREATE_TIME         | UPDATE_TIME | CHECK_TIME | TABLE_COLLATION | CHECKSUM | CREATE_OPTIONS | TABLE_COMMENT | TIDB_TABLE_ID | TIDB_ROW_ID_SHARDING_INFO | TIDB_PK_TYPE | TIDB_PLACEMENT_POLICY_NAME |
+---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
| def           | test         | t6613      | BASE TABLE | InnoDB |      10 | Compact    |          2 |              8 |          16 |               0 |           16 |         0 |              1 | 2024-07-25 21:10:37 | NULL        | NULL       | utf8mb4_bin     |     NULL |                |               |          8063 | NOT_SHARDED               | NONCLUSTERED | NULL                       |
+---------------+--------------+------------+------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+-----------------+----------+----------------+---------------+---------------+---------------------------+--------------+----------------------------+
1 row in set (0.00 sec)

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-07-25T14:37:46Z

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-07-25T15:15:12Z

Codecov Report

Attention: Patch coverage is 89.70588% with 28 lines in your changes missing coverage. Please review.

Project coverage is 74.6847%. Comparing base (199bf90) to head (f292a3c).
Report is 9 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #54918        +/-   ##
================================================
+ Coverage   72.6158%   74.6847%   +2.0688%     
================================================
  Files          1565       1565                
  Lines        440229     442705      +2476     
================================================
+ Hits         319676     330633     +10957     
+ Misses       100711      91766      -8945     
- Partials      19842      20306       +464

Flag	Coverage Δ
integration	`49.4718% <85.6617%> (?)`
unit	`71.7194% <87.1323%> (+0.0557%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9567% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`52.5020% <ø> (+6.6531%)`	⬆️

tiancaiamao · 2024-07-26T02:44:54Z

@tangenta

[2024/07/25 14:54:35.357 +00:00] [INFO] [schema_version.go:381] ["wait latest schema version changed(get the metadata lock if tidb_enable_metadata_lock is true)"] [category=ddl] [ver=52] ["take time"=2.815636ms] [job="ID:108, Type:create table, State:done, SchemaState:public, SchemaID:2, TableID:104, RowCount:0, ArgLen:2, start time: 2024-07-25 14:54:35.351 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]
[2024/07/25 14:54:35.359 +00:00] [INFO] [ddl_worker.go:940] ["finish DDL job"] [worker="worker 22, tp general"] [category=ddl] [jobID=108] [conn=3] [job="ID:108, Type:create table, State:synced, SchemaState:public, SchemaID:2, TableID:104, RowCount:0, ArgLen:0, start time: 2024-07-25 14:54:35.351 +0000 UTC, Err:<nil>, ErrCount:0, SnapshotVersion:0, LocalMode: false"]
[2024/07/25 14:54:35.361 +00:00] [INFO] [ddl.go:1271] ["DDL job is finished"] [category=ddl] [jobID=108]
    testkit.go:274: 
        	Error Trace:	pkg/testkit/testkit.go:274
        	            				pkg/testkit/testkit.go:199
        	            				pkg/testkit/testkit.go:174
        	            				pkg/planner/core/memtable_predicate_extractor_test.go:1679
        	Error:      	Received unexpected error:
        	            	[schema:1146]Table 'INFORMATION_SCHEMA.p' doesn't exist
        	            	github.com/pingcap/errors.AddStack
        	            		external/com_github_pingcap_errors/errors.go:178
        	            	github.com/pingcap/errors.Trace
        	            		external/com_github_pingcap_errors/juju_adaptor.go:15
        	            	github.com/pingcap/tidb/pkg/executor.getAllTableInfos
        	            		pkg/executor/infoschema_reader.go:588
        	            	github.com/pingcap/tidb/pkg/executor.(*memtableRetriever).setDataFromTables
        	            		pkg/executor/infoschema_reader.go:631
        	            	github.com/pingcap/tidb/pkg/executor.(*memtableRetriever).retrieve
        	            		pkg/executor/infoschema_reader.go:133
        	            	github.com/pingcap/tidb/pkg/executor.(*MemTableReaderExec).Next
        	            		pkg/executor/memtable_reader.go:120
        	            	github.com/pingcap/tidb/pkg/executor/internal/exec.Next
        	            		pkg/executor/internal/exec/executor.go:451
        	            	github.com/pingcap/tidb/pkg/executor.(*ExecStmt).next
        	            		pkg/executor/adapter.go:1249
        	            	github.com/pingcap/tidb/pkg/executor.(*recordSet).Next
        	            		pkg/executor/adapter.go:169
        	            	github.com/pingcap/tidb/pkg/session.GetRows4Test
        	            		pkg/session/tidb.go:351
        	            	github.com/pingcap/tidb/pkg/session.ResultSetToStringSlice
        	            		pkg/session/tidb.go:369
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).ResultSetToResultWithCtx
        	            		pkg/testkit/testkit.go:273
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQueryWithContext
        	            		pkg/testkit/testkit.go:199
        	            	github.com/pingcap/tidb/pkg/testkit.(*TestKit).MustQuery
        	            		pkg/testkit/testkit.go:174
        	            	github.com/pingcap/tidb/pkg/planner/core_test.TestTikvRegionStatusExtractor
        	            		pkg/planner/core/memtable_predicate_extractor_test.go:1679
        	            	testing.tRunner
        	            		GOROOT/src/testing/testing.go:1595
        	            	runtime.goexit
        	            		src/runtime/asm_amd64.s:1650
        	Test:       	TestTikvRegionStatusExtractor
        	Messages:   	sql:select * from information_schema.tables where table_name = 'p', args:[]
[2024/07/25 14:54:35.362 +00:00] [INFO] [mock.go:101] ["owner manager retire owner"] [ownerKey=/tidb/ddl/fg/owner] [ID=536f52c9-7647-4fea-bf30-15b953339b6d]
[2024/07/25 14:54:35.362 +00:00] [INFO] [job_table.go:268] ["scheduleLoop quit due to context canceled"] [category=ddl]

…by-name

tangenta · 2024-07-30T10:44:59Z

tests/integrationtest/r/infoschema/infoschema.result

@@ -115,6 +115,7 @@ Projection_4	8000.00	root		Column#5, Column#10
  └─MemTableScan_6	10000.00	root	table:TABLES	table_schema:["infoschema__infoschema"]
 select engine, DATA_LENGTH from information_schema.tables where lower(table_name) = 't5' and  upper(table_schema) = 'INFOSCHEMA__INFOSCHEMA';
 engine	DATA_LENGTH
+InnoDB	8


There is a bug in previous impl.

lance6716 · 2024-07-30T13:08:52Z

tests/integrationtest/t/infoschema/infoschema.test

+# TestPartitionsColumn
+create table pt1(a int primary key, b int) partition by hash(a) partitions 4;
+create table pt2(a int primary key, b int) partition by hash(a) partitions 4;
+-- sorted_result


I'm not familiar with these logic. Can these test make sure the execution does not query full data in TiKV?

No. It only make sure the result is correct. Let me add a manual test about information_schema.partitions.

lance6716

rest lgtm

pkg/planner/core/memtable_predicate_extractor.go

pkg/executor/infoschema_reader.go

ti-chi-bot · 2024-07-31T06:22:13Z

[LGTM Timeline notifier]

Timeline:

2024-07-30 07:01:20.314276687 +0000 UTC m=+252796.594324758: ☑️ agreed by tiancaiamao.
2024-07-31 06:22:12.211505647 +0000 UTC m=+336848.491553700: ☑️ agreed by lance6716.

tiprow · 2024-07-31T06:26:05Z

@tangenta: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`f175483`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot · 2024-07-31T07:38:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: AilinKid, lance6716, tiancaiamao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [AilinKid,lance6716,tiancaiamao]
~~pkg/planner/OWNERS~~ [AilinKid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/executor/infoschema_reader.go

tangenta · 2024-07-31T09:32:35Z

/retest

tiprow · 2024-07-31T09:32:56Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ref pingcap#50305

infoschema: adapt TableByName for query of memory table

96f9954

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 25, 2024

ti-chi-bot bot removed do-not-merge/needs-tests-checked do-not-merge/needs-linked-issue labels Jul 25, 2024

fix linter

3d7c4a5

tangenta added 4 commits July 26, 2024 15:42

fix table not exists error

518958c

refine code

597acb9

fix TestAlterDBPlacement

1eb0392

find table info by tidb_table_id

c5a97d2

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 29, 2024

tangenta added 3 commits July 30, 2024 10:24

Merge remote-tracking branch 'upstream/master' into HEAD

eb99174

fix integration test infoschema/infoschema

b67366e

adapt FindTableByPartitionID for information_schema.partitions

d4b8acb

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 30, 2024

tangenta added 2 commits July 30, 2024 12:16

fix linter

c011cd4

Merge remote-tracking branch 'upstream/master' into infoschema-adapt-…

4a41918

…by-name

tiancaiamao approved these changes Jul 30, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 30, 2024

tangenta added 2 commits July 30, 2024 17:43

fix integration test infoschema/infoschema

a7719ab

fix integration test infoschema/infoschema

5bd5a55

tangenta commented Jul 30, 2024

View reviewed changes

lance6716 reviewed Jul 30, 2024

View reviewed changes

lance6716 reviewed Jul 31, 2024

View reviewed changes

address comment

f175483

lance6716 approved these changes Jul 31, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 31, 2024

update bazel

f292a3c

AilinKid approved these changes Jul 31, 2024

View reviewed changes

ti-chi-bot bot added the approved label Jul 31, 2024

AilinKid reviewed Jul 31, 2024

View reviewed changes

pkg/executor/infoschema_reader.go Show resolved Hide resolved

ti-chi-bot bot merged commit 05853dd into pingcap:master Jul 31, 2024
21 checks passed

hawkingrei pushed a commit to hawkingrei/tidb that referenced this pull request Aug 1, 2024

infoschema: adapt TableByName for query of memory table (pingcap#54918)

75fc104

ref pingcap#50305

joechenrh added a commit to joechenrh/tidb that referenced this pull request Aug 2, 2024

Change the error code to adapt to pingcap#54918

2781e63

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infoschema: adapt TableByName for query of memory table #54918

infoschema: adapt TableByName for query of memory table #54918

tangenta commented Jul 25, 2024 •

edited

Loading

tiprow bot commented Jul 25, 2024

codecov bot commented Jul 25, 2024 •

edited

Loading

tiancaiamao commented Jul 26, 2024

tangenta Jul 30, 2024

lance6716 Jul 30, 2024

tangenta Jul 30, 2024

lance6716 left a comment

ti-chi-bot bot commented Jul 31, 2024

tiprow bot commented Jul 31, 2024

ti-chi-bot bot commented Jul 31, 2024

tangenta commented Jul 31, 2024

tiprow bot commented Jul 31, 2024

infoschema: adapt TableByName for query of memory table #54918

infoschema: adapt TableByName for query of memory table #54918

Conversation

tangenta commented Jul 25, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Jul 25, 2024

codecov bot commented Jul 25, 2024 • edited Loading

Codecov Report

tiancaiamao commented Jul 26, 2024

tangenta Jul 30, 2024

Choose a reason for hiding this comment

lance6716 Jul 30, 2024

Choose a reason for hiding this comment

tangenta Jul 30, 2024

Choose a reason for hiding this comment

lance6716 left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Jul 31, 2024

[LGTM Timeline notifier]

tiprow bot commented Jul 31, 2024

ti-chi-bot bot commented Jul 31, 2024

tangenta commented Jul 31, 2024

tiprow bot commented Jul 31, 2024

tangenta commented Jul 25, 2024 •

edited

Loading

codecov bot commented Jul 25, 2024 •

edited

Loading