* restore: fix alter auto increment id for no-primary-key table

Signed-off-by: 5kbpers <[email protected]>

* add integration test

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

Signed-off-by: Neil Shen <[email protected]>

* cmd: convert version command to flags

Signed-off-by: Neil Shen <[email protected]>

* address comments

Signed-off-by: Neil Shen <[email protected]>

Signed-off-by: Neil Shen <[email protected]>

* restore-util: Implement split/scatter (#274)

* implement split/scatter

Signed-off-by: 5kbpers <[email protected]>

* init test

Signed-off-by: 5kbpers <[email protected]>

* redesign output/input of the lib

Signed-off-by: 5kbpers <[email protected]>

* update dependency

Signed-off-by: 5kbpers <[email protected]>

* add commments and more tests

Signed-off-by: 5kbpers <[email protected]>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <[email protected]>

* fix potential data race

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* Apply suggestions from code review

Co-Authored-By: kennytm <[email protected]>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* update dependency

Signed-off-by: 5kbpers <[email protected]>

* resolve conflicts

Signed-off-by: 5kbpers <[email protected]>

* fix prefix rewrite

Signed-off-by: 5kbpers <[email protected]>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <[email protected]>

* fix test

Signed-off-by: 5kbpers <[email protected]>

* check if region has peer

Signed-off-by: 5kbpers <[email protected]>

* more logs

Signed-off-by: 5kbpers <[email protected]>

* restore-util: add split retry interval (#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <[email protected]>

* fix go.sum

Signed-off-by: 5kbpers <[email protected]>

* restore-util: wait for scatter region sequentially  (#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* restore-util: add on split hook (#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <[email protected]>

* Nil check onSplit

Co-Authored-By: kennytm <[email protected]>

* restore-util: fix returned new region is nil (#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <[email protected]>

* more logs

Signed-off-by: 5kbpers <[email protected]>

* *: gofmt

Signed-off-by: 5kbpers <[email protected]>

* Apply suggestions from code review

Co-Authored-By: kennytm <[email protected]>

* fix log

Signed-off-by: 5kbpers <[email protected]>

* restore-util: call onSplit on splitByRewriteRules (#285)

Signed-off-by: Neil Shen <[email protected]>

* restore-util: fix overlapped error message (#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <[email protected]>

* fix log message

Signed-off-by: 5kbpers <[email protected]>

* reduce error trace

Signed-off-by: 5kbpers <[email protected]>

* fix test

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* restore-util: log warning when cannot find matched rewrite rule (#299)

* restore-util: add method to set placement rules and store labels (#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <[email protected]>

* minor fix

Signed-off-by: disksing <[email protected]>

* address comment

Signed-off-by: disksing <[email protected]>

* add GetPlacementRules

Signed-off-by: disksing <[email protected]>

* fix test

Signed-off-by: disksing <[email protected]>

* restore-util: support batch split (#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <[email protected]>

* go fmt

Signed-off-by: 5kbpers <[email protected]>

* Apply suggestions from code review

Co-Authored-By: kennytm <[email protected]>

* address commits

Signed-off-by: 5kbpers <[email protected]>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <[email protected]>

* add onSplit callback

Signed-off-by: 5kbpers <[email protected]>

* fix test

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* restore-util: add upper bound time for waiting for scatter (#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <[email protected]>

* add log

Signed-off-by: 5kbpers <[email protected]>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* restore-util: fix wrong url (#306)

Signed-off-by: disksing <[email protected]>

* restore-util: add warning about unmatched table id (#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <[email protected]>

* fix log

Signed-off-by: 5kbpers <[email protected]>

* warn table id does not match

Signed-off-by: 5kbpers <[email protected]>

* add unit tests

Signed-off-by: 5kbpers <[email protected]>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <[email protected]>

* fix compile error

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* fix test

Signed-off-by: 5kbpers <[email protected]>

Co-authored-by: Ian <[email protected]>
Co-authored-by: Neil Shen <[email protected]>

* *: prune tidb-tools

Signed-off-by: Neil Shen <[email protected]>

* restore: address linters suggestions

Signed-off-by: Neil Shen <[email protected]>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <[email protected]>

* address comment

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: 5kbpers <[email protected]>
Co-authored-by: kennytm <[email protected]>
Co-authored-by: disksing <[email protected]>
Co-authored-by: Ian <[email protected]>

…tored (#148)

* Fixed handling for a dbName that do not exist in the backup being restored

* Fixed handling for a dbName that do not exist in the backup being restored

* validate: fix debug meta test ci

* *: extracts runBackup/runRestore in cmd into pkg/task

Defines a "Config" structure to store the parsed flags.

Use the "black-white-list" structure to define what tables/databases to
backup/restore.

* go.mod: update tidb to v4.0.0-beta

Co-authored-by: kennytm <[email protected]>

* restore: enhance error handling

Signed-off-by: 5kbpers <[email protected]>

* unit test

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* fix region epoch error

Signed-off-by: 5kbpers <[email protected]>

* address comments

Signed-off-by: 5kbpers <[email protected]>

* remove `Restore*`

Signed-off-by: 5kbpers <[email protected]>

* address lint

Signed-off-by: 5kbpers <[email protected]>

* add debug log

Signed-off-by: 5kbpers <[email protected]>

* Apply suggestions from code review

Co-Authored-By: kennytm <[email protected]>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <[email protected]>

* fix retry error

Signed-off-by: 5kbpers <[email protected]>

* handle RegionNotFound error

Signed-off-by: 5kbpers <[email protected]>

Co-authored-by: Neil Shen <[email protected]>
Co-authored-by: kennytm <[email protected]>