*: support REORGANIZE PARTITION, part 1 - data reorg / data+index copying #38460

mjonss · 2022-10-13T15:45:12Z

What problem does this PR solve?

First part of REORGANIZE PARTITION, implementing the data copying from one set of partitions to a new set of partitions
as in:
ALTER TABLE t REORGANIZE PARTITION INTO ()

This part covers:

Only this! Implementation of data copying from "list of partitions to replace" to "new partition definitions to move the data into".
No tests, *: Reorganize partition, rollback part 3 #38532 (part 3) includes both part 1 and part 2 ( *: Table partition double write during Reorganize partition (part 2) #38508 ) and adds the rest of the DDL code so it is complete and can be tested (part 3 also includes test for this PR).

Issue Number: ref #15000 , ref #38535

Problem Summary:

What is changed and how it works?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2022-10-13T15:45:14Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

bb7133
tangenta

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

mjonss · 2022-10-14T16:02:54Z

/run-all-test

mjonss · 2022-10-17T07:11:33Z

/run-build plugin=main

mjonss · 2022-10-17T07:14:56Z

/run-check_dev_2 plugin=main

wuhuizuo · 2022-10-17T07:34:43Z

/run-build plugin=master

mjonss · 2022-10-17T08:16:37Z

/run-build plugin=master

mjonss · 2022-10-17T11:28:48Z

/run-build

mjonss · 2022-10-17T11:28:58Z

/run-check_dev_2

mjonss · 2022-10-17T11:29:05Z

/run-unit-test

mjonss · 2022-10-20T12:52:45Z

/run-check_dev_2

mjonss · 2022-10-20T12:53:11Z

/run-unit-test

table/table.go

table/tables/partition.go

zimulala · 2022-11-14T03:03:12Z

table/tables/partition.go

@@ -939,19 +950,34 @@ func (t *partitionedTable) PartitionExpr() (*PartitionExpr, error) {
 	return t.partitionExpr, nil
 }

-func (t *partitionedTable) GetPartitionColumnNames() []model.CIStr {
+func (t *partitionedTable) GetPartitionColumnIDs() []int64 {


Do we need to add a UT for this function?

Since it is used by GetPartitionColumnNames, and that is included in one UT, I don't think it is needed. But if you like I can create such UT?

table/tables/partition.go

ddl/ddl_api.go

mjonss · 2022-11-15T14:19:45Z

/run-check_dev_2

ddl/partition.go

ddl/ddl_api.go

ddl/db_partition_test.go

ddl/ddl_api.go

table/tables/partition.go

close pingcap#38647

close pingcap#39346

…iable error (pingcap#39215) close pingcap#39214

ref pingcap#37887

close pingcap#37183

tangenta · 2022-12-22T13:21:11Z

ddl/backfilling.go

+			if err != nil {
+				if b.canSkipError(err) {
+					continue
+				}
+				return err
+			}


The errors returned from newReorgPartitionWorker are not recoverable... We had better not to skip them :)

OK, I agree and have removed it. But I don't think it would be too bad, since it would just limit the number of workers if by somehow the newReorgPartitionWorker would have succeeded.

When newReorgPartitionWorker is possible to succeed for part of workers(for example the memory is limited by the resource manager), we can add it back and update the log message inside canSkipError.

close pingcap#40111

…nto reorg-part-merge-dec22

…e-dec22

…nto reorg-part-data-reorg

mjonss · 2022-12-23T16:21:49Z

/run-unit-test

mjonss · 2022-12-24T09:53:42Z

/run-unit-test

ddl/column.go

bb7133

LGTM

Benjamin2037 · 2022-12-30T05:24:50Z

could we check that has ut option?

bb7133 · 2022-12-30T09:07:46Z

/merge

ti-chi-bot · 2022-12-30T09:07:51Z

This pull request has been accepted and is ready to merge.

Commit hash: 0eab90d

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 13, 2022

mjonss mentioned this pull request Oct 17, 2022

*: Table partition double write during Reorganize partition (part 2) #38508

Merged

12 tasks

mjonss mentioned this pull request Oct 30, 2022

Support REORGANIZE PARTITION Key Dev Task #38535

Closed

13 tasks

bb7133 closed this Nov 10, 2022

bb7133 reopened this Nov 10, 2022

zimulala reviewed Nov 14, 2022

View reviewed changes

bb7133 reviewed Nov 15, 2022

View reviewed changes

ddl/ddl_api.go Outdated Show resolved Hide resolved

bb7133 reviewed Nov 16, 2022

View reviewed changes

ddl/partition.go Outdated Show resolved Hide resolved

ddl/partition.go Outdated Show resolved Hide resolved

ddl/partition.go Outdated Show resolved Hide resolved

zimulala requested a review from tangenta November 17, 2022 07:39

tangenta reviewed Nov 21, 2022

View reviewed changes

Leavrth and others added 7 commits November 30, 2022 12:52

Backup: add checkpoint (pingcap#38648)

1de2bf6

close pingcap#38647

lightning: add precheck about downstream CDC/PiTR (pingcap#39338)

328aef8

close pingcap#39346

executor: add index_merge_intersection cases (pingcap#39323)

40f0bd5

executor: close resultset (pingcap#39474)

a48fe79

autoid_service,meta/autoid: distinguish retriable error from non-retr…

a00f0fe

…iable error (pingcap#39215) close pingcap#39214

Design: add the design doc for view hint (pingcap#39313)

31d943f

ref pingcap#37887

*: make auto-analyze concurrency configurable (pingcap#37078)

8e4748a

close pingcap#37183

ti-chi-bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Dec 22, 2022

Moved code to only run once instead of for every fetchRowColVal

1e0d902

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 22, 2022

tangenta reviewed Dec 22, 2022

View reviewed changes

Return directly on error instead of retrying workerCnt times

eb6fec9

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 22, 2022

crazycs520 and others added 4 commits December 22, 2022 23:02

ddl: fix issue of add foreign key too slow in big table (pingcap#40112)

1305687

close pingcap#40111

Merge remote-tracking branch 'pingcap/feature/reorganize-partition' i…

251a69a

…nto reorg-part-merge-dec22

Merge branch 'reorg-part-merge-dec22' into reorg-part-data-reorg-merg…

0a275d6

…e-dec22

Added reverted code needed for reorganize partition

2e5b311

tangenta approved these changes Dec 23, 2022

View reviewed changes

mjonss requested a review from a team as a code owner December 23, 2022 08:21

ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 23, 2022

Merge remote-tracking branch 'pingcap/feature/reorganize-partition' i…

0eab90d

…nto reorg-part-data-reorg

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 23, 2022

Benjamin2037 reviewed Dec 29, 2022

View reviewed changes

ddl/column.go Show resolved Hide resolved

bb7133 approved these changes Dec 29, 2022

View reviewed changes

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 30, 2022

ti-chi-bot merged commit 25df00e into pingcap:feature/reorganize-partition Dec 30, 2022

This was referenced Jan 31, 2023

*: (DNM) Reorganize partition #39161

Closed

*: REORGANIZE PARTITION (#38535) #41096

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: support REORGANIZE PARTITION, part 1 - data reorg / data+index copying #38460

*: support REORGANIZE PARTITION, part 1 - data reorg / data+index copying #38460

mjonss commented Oct 13, 2022 •

edited

Loading

ti-chi-bot commented Oct 13, 2022 •

edited

Loading

mjonss commented Oct 14, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 17, 2022

wuhuizuo commented Oct 17, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 20, 2022

mjonss commented Oct 20, 2022

zimulala Nov 14, 2022

mjonss Nov 14, 2022

mjonss commented Nov 15, 2022

tangenta Dec 22, 2022

mjonss Dec 22, 2022

tangenta Dec 23, 2022 •

edited

Loading

mjonss commented Dec 23, 2022

mjonss commented Dec 24, 2022

bb7133 left a comment

Benjamin2037 commented Dec 30, 2022

bb7133 commented Dec 30, 2022

ti-chi-bot commented Dec 30, 2022

*: support REORGANIZE PARTITION, part 1 - data reorg / data+index copying #38460

*: support REORGANIZE PARTITION, part 1 - data reorg / data+index copying #38460

Conversation

mjonss commented Oct 13, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Oct 13, 2022 • edited Loading

mjonss commented Oct 14, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 17, 2022

wuhuizuo commented Oct 17, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 17, 2022

mjonss commented Oct 20, 2022

mjonss commented Oct 20, 2022

zimulala Nov 14, 2022

Choose a reason for hiding this comment

mjonss Nov 14, 2022

Choose a reason for hiding this comment

mjonss commented Nov 15, 2022

tangenta Dec 22, 2022

Choose a reason for hiding this comment

mjonss Dec 22, 2022

Choose a reason for hiding this comment

tangenta Dec 23, 2022 • edited Loading

Choose a reason for hiding this comment

mjonss commented Dec 23, 2022

mjonss commented Dec 24, 2022

bb7133 left a comment

Choose a reason for hiding this comment

Benjamin2037 commented Dec 30, 2022

bb7133 commented Dec 30, 2022

ti-chi-bot commented Dec 30, 2022

mjonss commented Oct 13, 2022 •

edited

Loading

ti-chi-bot commented Oct 13, 2022 •

edited

Loading

tangenta Dec 23, 2022 •

edited

Loading