ddl: implement the core for multi-schema change #35429

tangenta · 2022-06-16T04:22:56Z

What problem does this PR solve?

Issue Number: ref #14766

Problem Summary:

What is changed and how it works?

This PR implements

the core of multi-schema change according to the RFC.
ADD COLUMN for multi-schema change

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2022-06-16T04:22:58Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

crazycs520
wjhuang2016
zimulala

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ddl/db_integration_test.go

ddl/column.go

ddl/db_change_test.go

ddl/ddl_api.go

ddl/index_modify_test.go

ddl/multi_schema_change.go

parser/model/ddl.go

ddl/ddl_api.go

ddl/column.go

ddl/multi_schema_change.go

ddl/ddl.go

ddl/column.go

parser/model/ddl.go

ddl/multi_schema_change.go

parser/model/ddl.go

ddl/multi_schema_change.go

sre-bot · 2022-06-22T01:17:49Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/fc53f91a5cb23f414d7d23321f29fd057c68c150

zimulala · 2022-06-23T01:21:44Z

parser/model/ddl.go

+	sub.SnapshotVer = proxyJob.SnapshotVer
+	sub.Args = proxyJob.Args
+	sub.State = proxyJob.State
+	sub.Warning = proxyJob.Warning


How about sub Error?

SubJob doesn't have an Error field. The error of proxy job is handled by handleRevertibleException.

proxyJob := sub.ToProxyJob(job) ver, err = w.runDDLJob(d, t, proxyJob) sub.FromProxyJob(proxyJob) handleRevertibleException(job, sub, proxyJob.Error)

zimulala · 2022-06-23T01:32:56Z

ddl/column.go

+	colInfo.ID = allocateColumnID(tblInfo)
+	colInfo.State = model.StateNone
+	// To support add column asynchronous, we should mark its offset as the last column.
+	// So that we can use origin column offset to get value from row.
+	colInfo.Offset = len(cols)
+	// Append the column info to the end of the tblInfo.Columns.
+	// It will reorder to the right offset in "Columns" when it state change to public.
+	tblInfo.Columns = append(cols, colInfo)


It is duplicated with line106-114? Could we use it for line106-114?

Yes. For now, line106-114 is only used by onAddColumns()(the plural version), which will be removed in the following PRs.

zimulala · 2022-06-23T02:00:50Z

ddl/multi_schema_change.go

+		}
+
+		// Save table info and sub-jobs for rolling back.
+		tblInfo, _ := t.GetTable(job.SchemaID, job.TableID)


Did we forget to check the error here?

zimulala · 2022-06-23T02:18:23Z

ddl/multi_schema_change.go

+				handleRevertibleException(job, sub, proxyJob.Error)
+				// The TableInfo and sub-jobs should be restored
+				// because some schema changes update the transaction aggressively.
+				return updateVersionAndTableInfo(d, t, job, tblInfo, true)


This function is executed on every state of every subJob. In this case, GenSchemaVersion is executed one more time for each subJob state. Why do we need to call it here?

Only when there is an error, this function is executed.

Or can we just do 'txn.Reset' here?

zimulala · 2022-06-23T03:16:19Z

ddl/multi_schema_change.go

+			}
+		}
+		// All the sub-jobs are non-revertible.
+		job.MultiSchemaInfo.Revertible = false


In one place we use MarkNonRevertible, here we set Revertible to false directly. I think we'd better unify them.

ddl/multi_schema_change.go

zimulala · 2022-06-23T03:34:31Z

ddl/multi_schema_change.go

+func onMultiSchemaChange(w *worker, d *ddlCtx, t *meta.Meta, job *model.Job) (ver int64, err error) {
+	if job.MultiSchemaInfo.Revertible {
+		// Handle the rolling back job.
+		if job.IsRollingback() {


Is our rollback test the next PR?

Also, should we add some precise tests that test the behavior of each subJob and each of its states?

The TestCancel is updated for the rollback test. Moreover, the tests about cancelling adding columns will be introduced in the next PR.

Also, should we add some precise tests that test the behavior of each subJob and each of its states?

Yes. This will be added to the test-infra.

zimulala

LGTM

crazycs520 · 2022-06-23T08:00:37Z

/run-all-tests

xhebox · 2022-06-24T05:57:04Z

/run-all-tests

xhebox · 2022-06-24T06:30:03Z

/merge

ti-chi-bot · 2022-06-24T06:30:07Z

This pull request has been accepted and is ready to merge.

Commit hash: 1a1ef47

sre-bot · 2022-06-24T12:05:27Z

TiDB MergeCI notify

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci/integration-cdc-test	🟢 all 35 tests passed	25 min	Existing passed
idc-jenkins-ci-tidb/integration-common-test	🟢 all 11 tests passed	19 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 12 tests passed	12 min	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	8 min 42 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	7 min 24 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	5 min 32 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	5 min 23 sec	Existing passed
idc-jenkins-ci-tidb/mybatis-test	🟢 all 1 tests passed	3 min 25 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	3 min 13 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

* upstream/master: (38 commits) planner: disable collate clause support for enum or set column (pingcap#35684) *: Provide a util to "pause" session in uint test (pingcap#35529) ddl: implement the core for multi-schema change (pingcap#35429) parser: XXXDatabaseStmt now use CIStr for DB name (pingcap#35668) *: remove real tikv test on github actions (pingcap#35710) lightning: support disable scheduler by key range (pingcap#34130) Revert "planner: fix the wrong cost formula of MPPExchanger" (pingcap#35707) ddl, owner: refresh ddl sequence cache after becoming the owner (pingcap#35516) store/copr: fix tiflash integration test by disable paging on tiflash (pingcap#35674) planner: new cost formula for IndexJoin (pingcap#35671) BR: fix backup ranges total time summary log (pingcap#35552) raw_restore: fix the issue that raw restore rewrite the t prefix keys (pingcap#35641) table: fix cannot find the temp table when it's created in a transaction (pingcap#35663) metrics: replace histogram with counter for DistSQLCoprCache (pingcap#35514) *: enable exportloopref (pingcap#35680) types: fix bad error message for numeric conversion issue#32744 (pingcap#34047) *: enable ineffassign (pingcap#35670) config, sysvar: map `max-server-connections` to `max_connections` (pingcap#35453) *: enable durationcheck for nogo linter (pingcap#35661) *: fix issue of cast string as time is incompatible with MySQL (pingcap#35455) ...

* upstream/master: (57 commits) types: fix incompatible implementation of jsonpath extraction (pingcap#35320) planner: fix TRACE PLAN TARGET = 'estimation' panic when meeting partition table (pingcap#35743) *: Add `testfork.RunTest` to run multiple tests in one function (pingcap#35746) sessionctx/variable: add tests to ensure skipInit can be removed (pingcap#35703) helper: request another PD if one of them is unavailable (pingcap#35750) metrics: add cached table related metrics to grafana panel (pingcap#34718) expression: use cloned RetType at `evaluateExprWithNull` when it may be changed. (pingcap#35759) executor: fix left join on partition table generate invalid lock key (pingcap#35732) readme: remove adopters (pingcap/docs#8725) (pingcap#35124) *: only add default value for final aggregation to fix the aggregate push down (partition) union case (pingcap#35443) planner: fix the wrong cost formula of MPPExchanger on cost model ver2 (pingcap#35718) sessionctx: support encoding and decoding statement context (pingcap#35688) txn: refactor ts acquisition within build and execute phases (pingcap#35376) ddl: for schema-level DDL method parameter is now XXXStmt (pingcap#35722) *: enable gofmt (pingcap#35721) planner: disable collate clause support for enum or set column (pingcap#35684) *: Provide a util to "pause" session in uint test (pingcap#35529) ddl: implement the core for multi-schema change (pingcap#35429) parser: XXXDatabaseStmt now use CIStr for DB name (pingcap#35668) *: remove real tikv test on github actions (pingcap#35710) ...

ddl: add framework for multi-schema change

0aa52df

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 16, 2022

hawkingrei and others added 2 commits June 16, 2022 12:31

Merge branch 'master' into msc-framework

dd93dcc

fix integration test

29e9a22

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 16, 2022

xiongjiwei reviewed Jun 16, 2022

View reviewed changes

ddl/db_integration_test.go Show resolved Hide resolved

tangenta requested review from zimulala, wjhuang2016 and xiongjiwei June 16, 2022 08:15

tangenta added the compatibility-breaker Violation of forwards/backwards compatibility in a design-time piece. label Jun 16, 2022

tangenta and others added 2 commits June 16, 2022 16:21

fix integration test

acae418

Merge branch 'master' into msc-framework

ded9c2e

wjhuang2016 reviewed Jun 17, 2022

View reviewed changes

tangenta added 2 commits June 17, 2022 16:13

rename and add comments

8a45600

add a comment to model.Job

31e13d4

wjhuang2016 reviewed Jun 17, 2022

View reviewed changes

ddl/ddl_api.go Show resolved Hide resolved

xiongjiwei reviewed Jun 19, 2022

View reviewed changes

ddl/column.go Show resolved Hide resolved

add rollback handling to fix integration test

306465d

zimulala reviewed Jun 20, 2022

View reviewed changes

parser/model/ddl.go Show resolved Hide resolved

ddl/multi_schema_change.go Outdated Show resolved Hide resolved

ddl/multi_schema_change.go Outdated Show resolved Hide resolved

crazycs520 reviewed Jun 20, 2022

View reviewed changes

parser/model/ddl.go Outdated Show resolved Hide resolved

tangenta added 2 commits June 20, 2022 21:21

address comment

91968e4

add a job sizeof test

7d2c42a

crazycs520 reviewed Jun 21, 2022

View reviewed changes

ddl/multi_schema_change.go Outdated Show resolved Hide resolved

crazycs520 reviewed Jun 21, 2022

View reviewed changes

ddl/multi_schema_change.go Show resolved Hide resolved

tangenta added 2 commits June 21, 2022 14:47

address comments

32880da

rename continuE to useMultiSchemaChange

3ca3ee1

zimulala reviewed Jun 23, 2022

View reviewed changes

address comment

285ab3a

zimulala added the require-LGT3 Indicates that the PR requires three LGTM. label Jun 23, 2022

remove tiny optimization to improve readability

9204d4b

zimulala approved these changes Jun 23, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jun 23, 2022

crazycs520 approved these changes Jun 23, 2022

View reviewed changes

ti-chi-bot added status/LGT3 The PR has already had 3 LGTM. and removed status/LGT2 Indicates that a PR has LGTM 2. labels Jun 23, 2022

Merge branch 'master' into msc-framework

1a1ef47

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 24, 2022

ti-chi-bot added 4 commits June 24, 2022 14:30

Merge branch 'master' into msc-framework

fc2c4d3

Merge branch 'master' into msc-framework

46776ac

Merge branch 'master' into msc-framework

58f1cdb

Merge branch 'master' into msc-framework

fc53f91

ti-chi-bot merged commit 2c4d1df into pingcap:master Jun 24, 2022

tangenta mentioned this pull request Jun 30, 2022

ddl: remove onDropColumns and onAddColumns #35862

Merged

12 tasks

tangenta mentioned this pull request Jul 28, 2022

Support changing multiple schema in ALTER TABLE #14766

Closed

39 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ddl: implement the core for multi-schema change #35429

ddl: implement the core for multi-schema change #35429

tangenta commented Jun 16, 2022

ti-chi-bot commented Jun 16, 2022 •

edited

Loading

sre-bot commented Jun 22, 2022 •

edited

Loading

zimulala Jun 23, 2022

tangenta Jun 23, 2022

zimulala Jun 23, 2022

tangenta Jun 23, 2022

zimulala Jun 23, 2022

zimulala Jun 23, 2022 •

edited

Loading

tangenta Jun 23, 2022

zimulala Jun 23, 2022

zimulala Jun 23, 2022

zimulala Jun 23, 2022

zimulala Jun 23, 2022

tangenta Jun 23, 2022

zimulala left a comment

crazycs520 commented Jun 23, 2022

xhebox commented Jun 24, 2022

xhebox commented Jun 24, 2022

ti-chi-bot commented Jun 24, 2022

sre-bot commented Jun 24, 2022

ddl: implement the core for multi-schema change #35429

ddl: implement the core for multi-schema change #35429

Conversation

tangenta commented Jun 16, 2022

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Jun 16, 2022 • edited Loading

sre-bot commented Jun 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zimulala Jun 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zimulala left a comment

Choose a reason for hiding this comment

crazycs520 commented Jun 23, 2022

xhebox commented Jun 24, 2022

xhebox commented Jun 24, 2022

ti-chi-bot commented Jun 24, 2022

sre-bot commented Jun 24, 2022

TiDB MergeCI notify

ti-chi-bot commented Jun 16, 2022 •

edited

Loading

sre-bot commented Jun 22, 2022 •

edited

Loading

zimulala Jun 23, 2022 •

edited

Loading