*: remove `mock.NewContext()` usage when building table meta in production code #56348

lcwangchao · 2024-09-26T13:31:37Z

What problem does this PR solve?

Issue Number: ref #53388

What changed and how does it work?

This PR removes the mock.Context usage in production codes when building table meta, including:

The codes that build system table meta.
The codes in lightning to build table meta from create table statement.
The codes in cmd/importer

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Use the below test to output the system table's meta in json format:

func TestDumpInitTableInfo(t *testing.T) {
	_, do := testkit.CreateMockStoreAndDomain(t)
	var sb strings.Builder
	is := do.InfoSchema()
	dbNames := is.AllSchemaNames()
	slices.SortFunc(dbNames, func(a, b pmodel.CIStr) int {
		return strings.Compare(a.L, b.L)
	})
	for _, db := range dbNames {
		sb.WriteString("database: " + db.O + "\n")
		infos, err := is.SchemaTableInfos(context.TODO(), db)
		require.NoError(t, err)
		slices.SortFunc(infos, func(a, b *model.TableInfo) int {
			return strings.Compare(a.Name.L, b.Name.L)
		})
		for _, info := range infos {
			// set UpdateTS to 0 to make the output stable.
			info.UpdateTS = 0
			if db.L == "metrics_schema" {
				// metrics_schema's ID is not stable, so we set it to 0.
				info.ID = 0
			}
			str, err := json.MarshalIndent(info, "", "\t")
			require.NoError(t, err)
			sb.WriteString(string(str))
			sb.WriteString("\n")
		}
	}
	require.NoError(t, os.WriteFile("/tmp/meta.json", []byte(sb.String()), 0644))
}

And try to diff the output file in master and this PR, there is no diff. So the behavior is the same when building system table meta.

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-09-26T13:31:56Z

Hi @lcwangchao. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-26T13:34:32Z

pkg/infoschema/perfschema/init.go

@@ -62,6 +64,7 @@ func Init() {
 				c.ID = int64(i) + 1
 			}
 			meta.DBID = dbID
+			meta.State = model.StatePublic


I think the master code lost this line to set the table state to public... Though it still works well without this state, I still fix it to make the meta complete.

codecov · 2024-09-26T13:43:40Z

Codecov Report

Attention: Patch coverage is 88.88889% with 2 lines in your changes missing coverage. Please review.

Project coverage is 76.1041%. Comparing base (65d740f) to head (8abc07a).
Report is 1 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #56348        +/-   ##
================================================
+ Coverage   73.3813%   76.1041%   +2.7228%     
================================================
  Files          1623       1645        +22     
  Lines        447183     455176      +7993     
================================================
+ Hits         328149     346408     +18259     
+ Misses        98920      87175     -11745     
- Partials      20114      21593      +1479

Flag	Coverage Δ
integration	`51.8855% <88.2352%> (?)`
unit	`72.5331% <100.0000%> (+0.0537%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9478% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.0417% <ø> (+17.5108%)`	⬆️

…ction code

lcwangchao · 2024-09-26T13:46:42Z

pkg/util/dbutil/dbutiltest/utils.go

 )

-// GetTableInfo returns table information.
-func GetTableInfo(ctx context.Context, db dbutil.QueryExecutor, schemaName string, tableName string) (*model.TableInfo, error) {


Remove GetTableInfo because it is not used by any code...

used by tiflow😓

lcwangchao · 2024-09-26T15:06:08Z

/retest

tiprow · 2024-09-26T15:06:30Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-29T02:03:36Z

lightning/pkg/importer/get_pre_info.go

 	createTableStmt, ok := astNode.(*ast.CreateTableStmt)
 	if !ok {
 		return nil, errors.New("cannot transfer the parsed SQL as an CREATE TABLE statement")
 	}
-	info, err := ddl.MockTableInfo(sctx, createTableStmt, tableID)
+	info, err := ddl.BuildTableInfoFromAST(metabuild.NewNonStrictContext(), createTableStmt)


Seems newTableInfo always use 0 as table it and it is not actually used when importing data:

tidb/lightning/pkg/importer/get_pre_info.go

Line 411 in b427e33

theTableInfo, err := newTableInfo(createTblSQL, 0)

PTAL @lance6716

D3Hunter · 2024-09-29T02:38:10Z

cmd/importer/parser.go

@@ -237,7 +237,8 @@ func parseTable(t *table, stmt *ast.CreateTableStmt) error {
 	t.name = stmt.Table.Name.L
 	t.columns = make([]*column, 0, len(stmt.Cols))

-	mockTbl, err := ddl.MockTableInfo(mock.NewContext(), stmt, 1)
+	mockTbl, err := ddl.BuildTableInfoFromAST(metabuild.NewNonStrictContext(), stmt)
+	mockTbl.ID = 1


what's this for?

It seems like an old tool to import data to tidb. But I think no one uses it now...

ti-chi-bot · 2024-09-29T02:41:23Z

[LGTM Timeline notifier]

Timeline:

2024-09-29 02:23:49.062555934 +0000 UTC m=+148784.482768945: ☑️ agreed by lance6716.
2024-09-29 02:41:22.269389494 +0000 UTC m=+149837.689602505: ☑️ agreed by D3Hunter.

YangKeao

LGTM

ti-chi-bot · 2024-09-29T07:00:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, hawkingrei, lance6716, qw4990, tangenta, YangKeao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [D3Hunter,YangKeao,hawkingrei,lance6716,qw4990,tangenta]
~~lightning/OWNERS~~ [D3Hunter,lance6716]
~~pkg/ddl/OWNERS~~ [D3Hunter,lance6716,tangenta]
~~pkg/infoschema/OWNERS~~ [D3Hunter,tangenta]
~~pkg/meta/OWNERS~~ [D3Hunter,lance6716,tangenta]
~~pkg/planner/OWNERS~~ [hawkingrei,qw4990]
~~pkg/table/OWNERS~~ [tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lcwangchao · 2024-09-29T07:48:00Z

/retest

tiprow · 2024-09-29T07:48:25Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lance6716 · 2024-09-29T09:15:16Z

/retest

tiprow · 2024-09-29T09:15:37Z

@lance6716: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-29T10:08:29Z

/retest

tiprow · 2024-09-29T10:08:50Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-29T10:55:42Z

/retest

tiprow · 2024-09-29T10:56:03Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao · 2024-09-29T11:53:55Z

/retest

tiprow · 2024-09-29T11:54:16Z

@lcwangchao: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lcwangchao force-pushed the removemockctxintablebuild branch from 894c162 to ba456b8 Compare September 26, 2024 13:31

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. sig/planner SIG: Planner labels Sep 26, 2024

lcwangchao mentioned this pull request Sep 26, 2024

remove mock.Context usage in production code #53388

Closed

lcwangchao commented Sep 26, 2024

View reviewed changes

*: remove mock.NewContext() usage when building table meta in produ…

a8c3edd

…ction code

lcwangchao force-pushed the removemockctxintablebuild branch from ba456b8 to a8c3edd Compare September 26, 2024 13:44

ti-chi-bot bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 26, 2024

lcwangchao commented Sep 26, 2024

View reviewed changes

lcwangchao commented Sep 29, 2024

View reviewed changes

lcwangchao requested review from D3Hunter, lance6716 and YangKeao September 29, 2024 02:11

lance6716 approved these changes Sep 29, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 29, 2024

D3Hunter approved these changes Sep 29, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 29, 2024

tangenta approved these changes Sep 29, 2024

View reviewed changes

hawkingrei approved these changes Sep 29, 2024

View reviewed changes

ti-chi-bot bot added the approved label Sep 29, 2024

qw4990 approved these changes Sep 29, 2024

View reviewed changes

Merge branch 'master' into removemockctxintablebuild

aaec705

YangKeao approved these changes Sep 29, 2024

View reviewed changes

Merge branch 'master' into removemockctxintablebuild

8abc07a

ti-chi-bot bot merged commit 8f0baf4 into pingcap:master Sep 29, 2024
23 checks passed

lcwangchao deleted the removemockctxintablebuild branch September 29, 2024 15:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: remove `mock.NewContext()` usage when building table meta in production code #56348

*: remove `mock.NewContext()` usage when building table meta in production code #56348

lcwangchao commented Sep 26, 2024 •

edited

Loading

tiprow bot commented Sep 26, 2024

lcwangchao Sep 26, 2024

codecov bot commented Sep 26, 2024 •

edited

Loading

lcwangchao Sep 26, 2024

D3Hunter Oct 14, 2024

lcwangchao commented Sep 26, 2024

tiprow bot commented Sep 26, 2024

lcwangchao Sep 29, 2024

D3Hunter Sep 29, 2024

lcwangchao Sep 29, 2024

ti-chi-bot bot commented Sep 29, 2024

YangKeao left a comment

ti-chi-bot bot commented Sep 29, 2024

lcwangchao commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

lance6716 commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

lcwangchao commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

lcwangchao commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

lcwangchao commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

*: remove mock.NewContext() usage when building table meta in production code #56348

*: remove mock.NewContext() usage when building table meta in production code #56348

Conversation

lcwangchao commented Sep 26, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Sep 26, 2024

lcwangchao Sep 26, 2024

Choose a reason for hiding this comment

codecov bot commented Sep 26, 2024 • edited Loading

Codecov Report

lcwangchao Sep 26, 2024

Choose a reason for hiding this comment

D3Hunter Oct 14, 2024

Choose a reason for hiding this comment

lcwangchao commented Sep 26, 2024

tiprow bot commented Sep 26, 2024

lcwangchao Sep 29, 2024

Choose a reason for hiding this comment

D3Hunter Sep 29, 2024

Choose a reason for hiding this comment

lcwangchao Sep 29, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Sep 29, 2024

[LGTM Timeline notifier]

YangKeao left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Sep 29, 2024

lcwangchao commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

lance6716 commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

lcwangchao commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

lcwangchao commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

lcwangchao commented Sep 29, 2024

tiprow bot commented Sep 29, 2024

*: remove `mock.NewContext()` usage when building table meta in production code #56348

*: remove `mock.NewContext()` usage when building table meta in production code #56348

lcwangchao commented Sep 26, 2024 •

edited

Loading

codecov bot commented Sep 26, 2024 •

edited

Loading