restore: remove tiflash replica before restore #194

5kbpers · 2020-03-17T05:44:25Z

Signed-off-by: 5kbpers [email protected]

What problem does this PR solve?

Support restoring to a cluster containing TiFlash, see #193

Check List

Tests

Integration test

Code changes

Has exported function/method change

Side effects

Possible performance regression
Increased code complexity

Signed-off-by: 5kbpers <[email protected]>

3pointer · 2020-03-17T05:51:07Z

pkg/restore/db.go

+		"alter table %s set tiflash replica 0",
+		utils.EncloseName(tableInfo.Name.O),
+	)
+	err = db.se.Execute(ctx, removeTiFlashSQL)


what if ignore this error when tidb not support set tiflash replica

Would fail, do we need to support the versions of tidb which don't support tiflash?

overvenus · 2020-03-17T06:02:14Z

Should change GetAllTiKVStore on restore, it fails if there is a tiflash node.

Signed-off-by: 5kbpers <[email protected]>

3pointer · 2020-03-17T09:36:03Z

Do we need add replica back after restore?

pkg/restore/client.go

overvenus · 2020-03-17T10:08:51Z

/run-all-tests

overvenus · 2020-03-17T10:12:06Z

pkg/restore/client.go

+}
+
+// GetTiFlashStores returns an id list of tiflash stores.
+func (rc *Client) GetTiFlashStores() ([]uint64, error) {


Could you add an option to GatAllTiKVStores to return TiFlash store only?

overvenus · 2020-03-17T10:22:32Z

pkg/restore/split.go

+	retryTimes := ctx.Value(retryTimes).(int)
+	if retryTimes > 10 {
+		log.Warn("get region info", zap.Stringer("region", regionInfo.Region))
+	}


Could you handle the retry outside of the method?

No, the region info would be printed here.

pkg/restore/client.go

Signed-off-by: 5kbpers <[email protected]>

3pointer · 2020-03-17T12:59:42Z

go.mod

 	github.com/pingcap/log v0.0.0-20200117041106-d28c14d3b1cd
 	github.com/pingcap/parser v0.0.0-20200305120128-bde9faa0df84
+	github.com/pingcap/pd v2.1.19+incompatible


should remove this?

pkg/restore/client.go

Signed-off-by: 5kbpers <[email protected]>

3pointer · 2020-03-17T14:31:21Z

/run-integration-test pd=release-4.0 tidb=release-4.0 tikv=release-4.0

Signed-off-by: 5kbpers <[email protected]>

codecov · 2020-03-17T14:50:36Z

Codecov Report

Merging #194 into master will decrease coverage by 1.89%.
The diff coverage is 53.87%.

@@            Coverage Diff             @@
##           master     #194      +/-   ##
==========================================
- Coverage   70.90%   69.00%   -1.90%     
==========================================
  Files          47       47              
  Lines        4389     4633     +244     
==========================================
+ Hits         3112     3197      +85     
- Misses        856      996     +140     
- Partials      421      440      +19

Impacted Files	Coverage Δ
pkg/restore/split.go	`55.35% <10.52%> (-10.95%)`	⬇️
pkg/task/restore.go	`48.21% <16.00%> (-9.09%)`	⬇️
pkg/restore/db.go	`48.64% <16.66%> (-11.86%)`	⬇️
pkg/restore/util.go	`78.38% <42.85%> (-1.18%)`	⬇️
pkg/task/restore_raw.go	`56.25% <56.25%> (ø)`
pkg/restore/client.go	`68.96% <59.73%> (-1.74%)`	⬇️
cmd/restore.go	`82.71% <66.66%> (-8.33%)`	⬇️
pkg/utils/pd.go	`66.66% <66.66%> (ø)`
pkg/restore/backoff.go	`75.00% <75.00%> (ø)`
pkg/restore/import.go	`66.53% <79.59%> (ø)`
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c3d26d9...338c4cf. Read the comment docs.

pkg/restore/client.go

overvenus

Rest LGTM

overvenus · 2020-03-17T15:56:10Z

pkg/utils/pd.go

+	"github.com/pingcap/errors"
+	"github.com/pingcap/pd/v4/server/schedule/placement"
+	"github.com/pingcap/tidb/tablecodec"
+	"github.com/tikv/client-go/codec"


To save build time, please use tidb/codec and do not import client-go for now.

overvenus · 2020-03-17T16:03:31Z

pkg/conn/conn.go

@@ -103,6 +103,9 @@ const (
 	// SkipTiFlash causes GetAllTiKVStores to skip the store when it is found to
 	// be a TiFlash node.
 	SkipTiFlash UnexpectedStoreBehavior = 1
+	// TiFlashOnly caused GetAllTiKVStores to skip the store which is not a
+	// TiFlash node.
+	TiFlashOnly UnexpectedStoreBehavior = 2


Could you rename UnexpectedStoreBehavior to StoreBehavior?

overvenus · 2020-03-17T16:07:00Z

cmd/restore.go

+
+	for _, d := range dbs {
+		for _, t := range d.Tables {
+			log.Info("get table", zap.Stringer("name", t.Info.Name), zap.Int("replica", t.TiFlashReplicas))


Could you cmd.Println the result?

Added a progress bar.

overvenus · 2020-03-17T16:11:55Z

pkg/restore/client.go

+	}
+	backendURL := storage.FormatBackendURL(rc.backend)
+	log.Info("update backup meta", zap.Stringer("path", &backendURL))
+	err = rc.storage.Write(rc.ctx, utils.MetaFile, backupMetaData)


Could you write to another file? Backup files are supposed to be immutable.

overvenus · 2020-03-17T16:13:12Z

pkg/restore/client.go

+	}
+
+	// Update backup meta
+	rc.backupMeta.Schemas = schemas


Can we skip write to storage if there is no TiFlash node?

overvenus · 2020-03-17T16:18:53Z

pkg/utils/pd.go

+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+	if resp.StatusCode == 412 {


412 feels like magic.

Signed-off-by: 5kbpers <[email protected]>

pkg/task/restore.go

3pointer · 2020-03-18T04:26:40Z

/run-integration-test

Co-Authored-By: 3pointer <[email protected]>

3pointer · 2020-03-18T05:13:28Z

/run-integration-test

5kbpers · 2020-03-18T05:31:03Z

/run-integration-test pd=release-4.0 tidb=release-4.0 tikv=release-4.0

overvenus

LGTM

3pointer

LGTM

* restore: merge tidb-tools/pkg/restore-util (#146) * restore-util: Implement split/scatter (#274) * implement split/scatter Signed-off-by: 5kbpers <[email protected]> * init test Signed-off-by: 5kbpers <[email protected]> * redesign output/input of the lib Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * add commments and more tests Signed-off-by: 5kbpers <[email protected]> * add ScanRegions interface to Client Signed-off-by: 5kbpers <[email protected]> * fix potential data race Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * resolve conflicts Signed-off-by: 5kbpers <[email protected]> * fix prefix rewrite Signed-off-by: 5kbpers <[email protected]> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * check if region has peer Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * restore-util: add split retry interval (#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <[email protected]> * fix go.sum Signed-off-by: 5kbpers <[email protected]> * restore-util: wait for scatter region sequentially (#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add on split hook (#281) * restore-util: add on split hook Signed-off-by: Neil Shen <[email protected]> * Nil check onSplit Co-Authored-By: kennytm <[email protected]> * restore-util: fix returned new region is nil (#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * *: gofmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * restore-util: call onSplit on splitByRewriteRules (#285) Signed-off-by: Neil Shen <[email protected]> * restore-util: fix overlapped error message (#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <[email protected]> * fix log message Signed-off-by: 5kbpers <[email protected]> * reduce error trace Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: log warning when cannot find matched rewrite rule (#299) * restore-util: add method to set placement rules and store labels (#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <[email protected]> * minor fix Signed-off-by: disksing <[email protected]> * address comment Signed-off-by: disksing <[email protected]> * add GetPlacementRules Signed-off-by: disksing <[email protected]> * fix test Signed-off-by: disksing <[email protected]> * restore-util: support batch split (#300) * restore-util: support batch split Signed-off-by: 5kbpers <[email protected]> * go fmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * address commits Signed-off-by: 5kbpers <[email protected]> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <[email protected]> * add onSplit callback Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add upper bound time for waiting for scatter (#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <[email protected]> * add log Signed-off-by: 5kbpers <[email protected]> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: fix wrong url (#306) Signed-off-by: disksing <[email protected]> * restore-util: add warning about unmatched table id (#313) * restore-util: support table partition Signed-off-by: 5kbpers <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * warn table id does not match Signed-off-by: 5kbpers <[email protected]> * add unit tests Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: Neil Shen <[email protected]> * fix compile error Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ian <[email protected]> Co-authored-by: Neil Shen <[email protected]> * *: prune tidb-tools Signed-off-by: Neil Shen <[email protected]> * restore: address linters suggestions Signed-off-by: Neil Shen <[email protected]> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <[email protected]> * address comment Signed-off-by: Neil Shen <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ian <[email protected]> * Fixed handling for a dbName that do not exist in the backup being restored (#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (#150) Co-authored-by: kennytm <[email protected]> * restore: enhance error handling (#152) * restore: enhance error handling Signed-off-by: 5kbpers <[email protected]> * unit test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix region epoch error Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * remove `Restore*` Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * add debug log Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * Update pkg/restore/import.go Co-Authored-By: kennytm <[email protected]> * fix retry error Signed-off-by: 5kbpers <[email protected]> * handle RegionNotFound error Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * Incremental BR: support DDL (#155) * support backup&restore ddl Signed-off-by: 5kbpers <[email protected]> * integration tests Signed-off-by: 5kbpers <[email protected]> * update kvproto Signed-off-by: 5kbpers <[email protected]> * fix integration tests Signed-off-by: 5kbpers <[email protected]> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * add unit test Signed-off-by: 5kbpers <[email protected]> * fix tests Signed-off-by: 5kbpers <[email protected]> * disable fast checksum in incremental br Signed-off-by: 5kbpers <[email protected]> * fix no valid key error Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Reduce TiDB dependencies (#158) * utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <[email protected]> * go.mod: update tidb (#168) Signed-off-by: Neil Shen <[email protected]> * BR support TLS (#161) * *: support tls * move tikv.driver to glue * fix comments * upgrade golangci and prepare for go 1.14 (#171) Signed-off-by: Neil Shen <[email protected]> * backup: add raw backup command (#101) * backup: add raw backup command * restore: speed up retry on not leader (#179) * tests: stable cluster start up Signed-off-by: Neil Shen <[email protected]> * tests: fix unbound var Signed-off-by: Neil Shen <[email protected]> * restore: speed retry on not leader Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * tests: add --cacert flag Signed-off-by: Neil Shen <[email protected]> * make codecov green Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * conn, restore: paginate scan regions (#165) * conn, restore: paginate scan regions Signed-off-by: Neil Shen <[email protected]> * tests: large timeout Signed-off-by: Neil Shen <[email protected]> * Batch restore (#167) * *: unify Range and RangeTree Signed-off-by: Neil Shen <[email protected]> * restore: split restore files into small batch Signed-off-by: Neil Shen <[email protected]> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <[email protected]> * restore: unused table worker pool Signed-off-by: Neil Shen <[email protected]> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <[email protected]> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <[email protected]> * README, docker: add quick start (#181) * README, docker: add quick start Signed-off-by: Neil Shen <[email protected]> * cmd: disable some TiDB log Signed-off-by: Neil Shen <[email protected]> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <[email protected]> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <[email protected]> * *: update tidb dependency build with go1.14 (#176) * *: add license header (#182) * rtree: move checkFile into backup Signed-off-by: Neil Shen <[email protected]> * *: add license header Signed-off-by: Neil Shen <[email protected]> * Update LICENSE.md Co-Authored-By: kennytm <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: 3pointer <[email protected]> * conn: support not shutting down the storage when closing the connection (#185) Co-authored-by: 3pointer <[email protected]> * conn: use GetDomain to avoid some TiDB breaking changes (#186) * conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <[email protected]> * minor usability improvement Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * fix check safepoint & unhide experimental features (#175) * backup: check safepoint for last backup ts Signed-off-by: 5kbpers <[email protected]> * check lastbackupts > 0 Signed-off-by: 5kbpers <[email protected]> * unhide experimental features Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <[email protected]> Co-authored-by: kennytm <[email protected]> * support backupts (#172) * support backupts * address comment * address comment * fix space * *: update pd deps to v4 (#184) Co-authored-by: 3pointer <[email protected]> * restore: support online restore (#114) Signed-off-by: disksing <[email protected]> * metrics: add grafana scripts (#140) * add grafana scripts * fix Co-authored-by: 3pointer <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: kennytm <[email protected]> * filter out all TiFlash nodes when retrieving lists of stores from PD (#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup * Create integration test for S3 storage (#174) * Fix summary log (#191) * *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <[email protected]> * Implement Raw Restore (#104) * Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <[email protected]> * Remove unnecessary comments Signed-off-by: MyonKeminta <[email protected]> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <[email protected]> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <[email protected]> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <[email protected]> * Fix tests Signed-off-by: MyonKeminta <[email protected]> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <[email protected]> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: Neil Shen <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> * restore: remove tiflash replica before restore (#194) * restore: remove tiflash replica before restore Signed-off-by: 5kbpers <[email protected]> * rename errSplit variable Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * check replica count by region info Signed-off-by: 5kbpers <[email protected]> * cleanup Signed-off-by: 5kbpers <[email protected]> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <[email protected]> * fix save crcxor Signed-off-by: 5kbpers <[email protected]> * fix decode the key of placement rule Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Update pkg/task/restore.go Co-Authored-By: 3pointer <[email protected]> Co-authored-by: 3pointer <[email protected]> * summary: put summary log at last (#197) * summary: put summary log at last * fix switch sql Co-authored-by: Neil Shen <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ian <[email protected]> Co-authored-by: Kolbe Kegel <[email protected]> Co-authored-by: WangXiangUSTC <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]>

* restore: fix alter auto increment id for no-primary-key table (#139) * restore: fix alter auto increment id for no-primary-key table Signed-off-by: 5kbpers <[email protected]> * add integration test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * *: use oracle package to manipulate ts and test gc safe point (#121) Signed-off-by: Neil Shen <[email protected]> * cmd: convert version command to flags (#144) * cmd: convert version command to flags Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * *: add changelog and github templates (#143) Signed-off-by: Neil Shen <[email protected]> * restore: merge tidb-tools/pkg/restore-util (#146) * restore-util: Implement split/scatter (#274) * implement split/scatter Signed-off-by: 5kbpers <[email protected]> * init test Signed-off-by: 5kbpers <[email protected]> * redesign output/input of the lib Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * add commments and more tests Signed-off-by: 5kbpers <[email protected]> * add ScanRegions interface to Client Signed-off-by: 5kbpers <[email protected]> * fix potential data race Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * resolve conflicts Signed-off-by: 5kbpers <[email protected]> * fix prefix rewrite Signed-off-by: 5kbpers <[email protected]> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * check if region has peer Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * restore-util: add split retry interval (#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <[email protected]> * fix go.sum Signed-off-by: 5kbpers <[email protected]> * restore-util: wait for scatter region sequentially (#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add on split hook (#281) * restore-util: add on split hook Signed-off-by: Neil Shen <[email protected]> * Nil check onSplit Co-Authored-By: kennytm <[email protected]> * restore-util: fix returned new region is nil (#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * *: gofmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * restore-util: call onSplit on splitByRewriteRules (#285) Signed-off-by: Neil Shen <[email protected]> * restore-util: fix overlapped error message (#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <[email protected]> * fix log message Signed-off-by: 5kbpers <[email protected]> * reduce error trace Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: log warning when cannot find matched rewrite rule (#299) * restore-util: add method to set placement rules and store labels (#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <[email protected]> * minor fix Signed-off-by: disksing <[email protected]> * address comment Signed-off-by: disksing <[email protected]> * add GetPlacementRules Signed-off-by: disksing <[email protected]> * fix test Signed-off-by: disksing <[email protected]> * restore-util: support batch split (#300) * restore-util: support batch split Signed-off-by: 5kbpers <[email protected]> * go fmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * address commits Signed-off-by: 5kbpers <[email protected]> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <[email protected]> * add onSplit callback Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add upper bound time for waiting for scatter (#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <[email protected]> * add log Signed-off-by: 5kbpers <[email protected]> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: fix wrong url (#306) Signed-off-by: disksing <[email protected]> * restore-util: add warning about unmatched table id (#313) * restore-util: support table partition Signed-off-by: 5kbpers <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * warn table id does not match Signed-off-by: 5kbpers <[email protected]> * add unit tests Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: Neil Shen <[email protected]> * fix compile error Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ian <[email protected]> Co-authored-by: Neil Shen <[email protected]> * *: prune tidb-tools Signed-off-by: Neil Shen <[email protected]> * restore: address linters suggestions Signed-off-by: Neil Shen <[email protected]> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <[email protected]> * address comment Signed-off-by: Neil Shen <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ian <[email protected]> * Fixed handling for a dbName that do not exist in the backup being restored (#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (#150) Co-authored-by: kennytm <[email protected]> * restore: enhance error handling (#152) * restore: enhance error handling Signed-off-by: 5kbpers <[email protected]> * unit test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix region epoch error Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * remove `Restore*` Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * add debug log Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * Update pkg/restore/import.go Co-Authored-By: kennytm <[email protected]> * fix retry error Signed-off-by: 5kbpers <[email protected]> * handle RegionNotFound error Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * Incremental BR: support DDL (#155) * support backup&restore ddl Signed-off-by: 5kbpers <[email protected]> * integration tests Signed-off-by: 5kbpers <[email protected]> * update kvproto Signed-off-by: 5kbpers <[email protected]> * fix integration tests Signed-off-by: 5kbpers <[email protected]> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * add unit test Signed-off-by: 5kbpers <[email protected]> * fix tests Signed-off-by: 5kbpers <[email protected]> * disable fast checksum in incremental br Signed-off-by: 5kbpers <[email protected]> * fix no valid key error Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Reduce TiDB dependencies (#158) * utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <[email protected]> * go.mod: update tidb (#168) Signed-off-by: Neil Shen <[email protected]> * BR support TLS (#161) * *: support tls * move tikv.driver to glue * fix comments * upgrade golangci and prepare for go 1.14 (#171) Signed-off-by: Neil Shen <[email protected]> * backup: add raw backup command (#101) * backup: add raw backup command * restore: speed up retry on not leader (#179) * tests: stable cluster start up Signed-off-by: Neil Shen <[email protected]> * tests: fix unbound var Signed-off-by: Neil Shen <[email protected]> * restore: speed retry on not leader Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * tests: add --cacert flag Signed-off-by: Neil Shen <[email protected]> * make codecov green Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * conn, restore: paginate scan regions (#165) * conn, restore: paginate scan regions Signed-off-by: Neil Shen <[email protected]> * tests: large timeout Signed-off-by: Neil Shen <[email protected]> * Batch restore (#167) * *: unify Range and RangeTree Signed-off-by: Neil Shen <[email protected]> * restore: split restore files into small batch Signed-off-by: Neil Shen <[email protected]> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <[email protected]> * restore: unused table worker pool Signed-off-by: Neil Shen <[email protected]> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <[email protected]> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <[email protected]> * README, docker: add quick start (#181) * README, docker: add quick start Signed-off-by: Neil Shen <[email protected]> * cmd: disable some TiDB log Signed-off-by: Neil Shen <[email protected]> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <[email protected]> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <[email protected]> * *: update tidb dependency build with go1.14 (#176) * *: add license header (#182) * rtree: move checkFile into backup Signed-off-by: Neil Shen <[email protected]> * *: add license header Signed-off-by: Neil Shen <[email protected]> * Update LICENSE.md Co-Authored-By: kennytm <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: 3pointer <[email protected]> * conn: support not shutting down the storage when closing the connection (#185) Co-authored-by: 3pointer <[email protected]> * conn: use GetDomain to avoid some TiDB breaking changes (#186) * conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <[email protected]> * minor usability improvement Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * fix check safepoint & unhide experimental features (#175) * backup: check safepoint for last backup ts Signed-off-by: 5kbpers <[email protected]> * check lastbackupts > 0 Signed-off-by: 5kbpers <[email protected]> * unhide experimental features Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <[email protected]> Co-authored-by: kennytm <[email protected]> * support backupts (#172) * support backupts * address comment * address comment * fix space * *: update pd deps to v4 (#184) Co-authored-by: 3pointer <[email protected]> * restore: support online restore (#114) Signed-off-by: disksing <[email protected]> * metrics: add grafana scripts (#140) * add grafana scripts * fix Co-authored-by: 3pointer <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: kennytm <[email protected]> * filter out all TiFlash nodes when retrieving lists of stores from PD (#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup * Create integration test for S3 storage (#174) * Fix summary log (#191) * *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <[email protected]> * Implement Raw Restore (#104) * Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <[email protected]> * Remove unnecessary comments Signed-off-by: MyonKeminta <[email protected]> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <[email protected]> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <[email protected]> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <[email protected]> * Fix tests Signed-off-by: MyonKeminta <[email protected]> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <[email protected]> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: Neil Shen <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> * update dep * restore: remove tiflash replica before restore (#194) * restore: remove tiflash replica before restore Signed-off-by: 5kbpers <[email protected]> * rename errSplit variable Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * check replica count by region info Signed-off-by: 5kbpers <[email protected]> * cleanup Signed-off-by: 5kbpers <[email protected]> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <[email protected]> * fix save crcxor Signed-off-by: 5kbpers <[email protected]> * fix decode the key of placement rule Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Update pkg/task/restore.go Co-Authored-By: 3pointer <[email protected]> Co-authored-by: 3pointer <[email protected]> * summary: put summary log at last (#197) * summary: put summary log at last * fix switch sql * *: abstract the progress channel (updateCh) into the glue package (#196) * *: abstract the progress channel (updateCh) into the glue package * restore: fix crash in truncateTS() when the bound is unlimited * task: fix comment Co-authored-by: Ian <[email protected]> * *: refline logs (#189) * adapt pd/tidb related 3.1 changes * tests: disable TLS test (#204) Signed-off-by: Neil Shen <[email protected]> * *: add S3 quick start and few enhancement of log (#202) * README, docker: add quick start about S3 storage Signed-off-by: Neil Shen <[email protected]> * pkg/summary: make sure to output correct summary Signed-off-by: Neil Shen <[email protected]> * cmd, tests: log to terminal if BR_LOG_TO_TERM is set Signed-off-by: Neil Shen <[email protected]> * Update pkg/task/common.go Co-Authored-By: kennytm <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * tests: cat log if br fails Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * restore: add error field to `DownloadResponse` (#195) * restore: add error field to `DownloadResponse` Signed-off-by: 5kbpers <[email protected]> * restore: populate restore cancel error (#207) Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * enhance usability of br (#208) * silenceUsage only when parse cmd flags failed * udpate tidb Co-authored-by: kennytm <[email protected]> * task: do not run checksum if restore failed (#209) * fix incremental bug in llroad test (#199) * restore: filter same table ddl * *: do not return error when backup/restore data is empty * fix create database double during incremental restore * add tests * fix ci * address comment * add skip create sqls (#211) * Revert "tests: disable TLS test (#204)" (#218) This reverts commit e168a60. * doc: add `minio` to dependence list. (#221) The README of test omitted `minio` in the dependence list, which is needed for run the integration test. Co-authored-by: Neil Shen <[email protected]> * move waiting reject stores in import file (#222) * move wait rejectstores into import files * restore: use new table id to search placementRules * Update pkg/restore/import.go Co-Authored-By: Neil Shen <[email protected]> * Update pkg/restore/import.go Co-Authored-By: kennytm <[email protected]> * fix ci Co-authored-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * Max index length (#220) * restore: set max-index-length to max * restore:add max-index-length params * address comment * address comment * glue: create schema/table directly with info (#216) * glue: create schema/table directly with info * go.mod: change to use the master version * gluetidb: fix failure to create schema * gluetidb: exclude non-public indices when restoring * go.mod: removed unused replace Co-authored-by: 3pointer <[email protected]> * update deps Co-authored-by: 5kbpers <[email protected]> Co-authored-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ian <[email protected]> Co-authored-by: Kolbe Kegel <[email protected]> Co-authored-by: WangXiangUSTC <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> Co-authored-by: 庄天翼 <[email protected]> Co-authored-by: 山岚 <[email protected]>

* restore: merge tidb-tools/pkg/restore-util (#146) * restore-util: Implement split/scatter (#274) * implement split/scatter Signed-off-by: 5kbpers <[email protected]> * init test Signed-off-by: 5kbpers <[email protected]> * redesign output/input of the lib Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * add commments and more tests Signed-off-by: 5kbpers <[email protected]> * add ScanRegions interface to Client Signed-off-by: 5kbpers <[email protected]> * fix potential data race Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * resolve conflicts Signed-off-by: 5kbpers <[email protected]> * fix prefix rewrite Signed-off-by: 5kbpers <[email protected]> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * check if region has peer Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * restore-util: add split retry interval (#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <[email protected]> * fix go.sum Signed-off-by: 5kbpers <[email protected]> * restore-util: wait for scatter region sequentially (#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add on split hook (#281) * restore-util: add on split hook Signed-off-by: Neil Shen <[email protected]> * Nil check onSplit Co-Authored-By: kennytm <[email protected]> * restore-util: fix returned new region is nil (#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * *: gofmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * restore-util: call onSplit on splitByRewriteRules (#285) Signed-off-by: Neil Shen <[email protected]> * restore-util: fix overlapped error message (#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <[email protected]> * fix log message Signed-off-by: 5kbpers <[email protected]> * reduce error trace Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: log warning when cannot find matched rewrite rule (#299) * restore-util: add method to set placement rules and store labels (#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <[email protected]> * minor fix Signed-off-by: disksing <[email protected]> * address comment Signed-off-by: disksing <[email protected]> * add GetPlacementRules Signed-off-by: disksing <[email protected]> * fix test Signed-off-by: disksing <[email protected]> * restore-util: support batch split (#300) * restore-util: support batch split Signed-off-by: 5kbpers <[email protected]> * go fmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * address commits Signed-off-by: 5kbpers <[email protected]> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <[email protected]> * add onSplit callback Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add upper bound time for waiting for scatter (#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <[email protected]> * add log Signed-off-by: 5kbpers <[email protected]> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: fix wrong url (#306) Signed-off-by: disksing <[email protected]> * restore-util: add warning about unmatched table id (#313) * restore-util: support table partition Signed-off-by: 5kbpers <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * warn table id does not match Signed-off-by: 5kbpers <[email protected]> * add unit tests Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: Neil Shen <[email protected]> * fix compile error Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ian <[email protected]> Co-authored-by: Neil Shen <[email protected]> * *: prune tidb-tools Signed-off-by: Neil Shen <[email protected]> * restore: address linters suggestions Signed-off-by: Neil Shen <[email protected]> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <[email protected]> * address comment Signed-off-by: Neil Shen <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ian <[email protected]> * Fixed handling for a dbName that do not exist in the backup being restored (#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored * validate: fix debug meta test ci (#153) * validate: fix debug meta test ci * *: extracts runBackup/runRestore in cmd into pkg/task (#156) * *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta * restore: fix restore summary log (#150) Co-authored-by: kennytm <[email protected]> * restore: enhance error handling (#152) * restore: enhance error handling Signed-off-by: 5kbpers <[email protected]> * unit test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix region epoch error Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * remove `Restore*` Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * add debug log Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * Update pkg/restore/import.go Co-Authored-By: kennytm <[email protected]> * fix retry error Signed-off-by: 5kbpers <[email protected]> * handle RegionNotFound error Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * Incremental BR: support DDL (#155) * support backup&restore ddl Signed-off-by: 5kbpers <[email protected]> * integration tests Signed-off-by: 5kbpers <[email protected]> * update kvproto Signed-off-by: 5kbpers <[email protected]> * fix integration tests Signed-off-by: 5kbpers <[email protected]> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * add unit test Signed-off-by: 5kbpers <[email protected]> * fix tests Signed-off-by: 5kbpers <[email protected]> * disable fast checksum in incremental br Signed-off-by: 5kbpers <[email protected]> * fix no valid key error Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Reduce TiDB dependencies (#158) * utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <[email protected]> * go.mod: update tidb (#168) Signed-off-by: Neil Shen <[email protected]> * BR support TLS (#161) * *: support tls * move tikv.driver to glue * fix comments * upgrade golangci and prepare for go 1.14 (#171) Signed-off-by: Neil Shen <[email protected]> * backup: add raw backup command (#101) * backup: add raw backup command * restore: speed up retry on not leader (#179) * tests: stable cluster start up Signed-off-by: Neil Shen <[email protected]> * tests: fix unbound var Signed-off-by: Neil Shen <[email protected]> * restore: speed retry on not leader Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * tests: add --cacert flag Signed-off-by: Neil Shen <[email protected]> * make codecov green Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * conn, restore: paginate scan regions (#165) * conn, restore: paginate scan regions Signed-off-by: Neil Shen <[email protected]> * tests: large timeout Signed-off-by: Neil Shen <[email protected]> * Batch restore (#167) * *: unify Range and RangeTree Signed-off-by: Neil Shen <[email protected]> * restore: split restore files into small batch Signed-off-by: Neil Shen <[email protected]> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <[email protected]> * restore: unused table worker pool Signed-off-by: Neil Shen <[email protected]> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <[email protected]> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <[email protected]> * README, docker: add quick start (#181) * README, docker: add quick start Signed-off-by: Neil Shen <[email protected]> * cmd: disable some TiDB log Signed-off-by: Neil Shen <[email protected]> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <[email protected]> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <[email protected]> * *: update tidb dependency build with go1.14 (#176) * *: add license header (#182) * rtree: move checkFile into backup Signed-off-by: Neil Shen <[email protected]> * *: add license header Signed-off-by: Neil Shen <[email protected]> * Update LICENSE.md Co-Authored-By: kennytm <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: 3pointer <[email protected]> * conn: support not shutting down the storage when closing the connection (#185) Co-authored-by: 3pointer <[email protected]> * conn: use GetDomain to avoid some TiDB breaking changes (#186) * conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <[email protected]> * minor usability improvement Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * fix check safepoint & unhide experimental features (#175) * backup: check safepoint for last backup ts Signed-off-by: 5kbpers <[email protected]> * check lastbackupts > 0 Signed-off-by: 5kbpers <[email protected]> * unhide experimental features Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <[email protected]> Co-authored-by: kennytm <[email protected]> * support backupts (#172) * support backupts * address comment * address comment * fix space * *: update pd deps to v4 (#184) Co-authored-by: 3pointer <[email protected]> * restore: support online restore (#114) Signed-off-by: disksing <[email protected]> * metrics: add grafana scripts (#140) * add grafana scripts * fix Co-authored-by: 3pointer <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: kennytm <[email protected]> * filter out all TiFlash nodes when retrieving lists of stores from PD (#187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup * Create integration test for S3 storage (#174) * Fix summary log (#191) * *: fix restore summary log after restore logic changed to files * fix * fix * fix Co-authored-by: kennytm <[email protected]> * Implement Raw Restore (#104) * Update kvproto * Implement raw restore * fix build * Set range for file importer Signed-off-by: MyonKeminta <[email protected]> * Remove unnecessary comments Signed-off-by: MyonKeminta <[email protected]> * check cf and support multi ranges in BackupMeta Signed-off-by: MyonKeminta <[email protected]> * Check files' cf; address comments * adjust structure to keep consistent with master * Fix build Signed-off-by: MyonKeminta <[email protected]> * Fix build and make check, avoid accessing TiDB in rawkv mode * Fix test Signed-off-by: MyonKeminta <[email protected]> * Fix tests Signed-off-by: MyonKeminta <[email protected]> * Fix broken logic after merging master * Update pkg/task/restore_raw.go Co-Authored-By: Neil Shen <[email protected]> * Address comments * Address comments * Mark raw restore as experimental * Fix build * Address comments * test: Add check for deleting data and partial backup * Fix build * Add license header * fix ci * fix ci Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: 3pointer <[email protected]> Co-authored-by: Neil Shen <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> * restore: remove tiflash replica before restore (#194) * restore: remove tiflash replica before restore Signed-off-by: 5kbpers <[email protected]> * rename errSplit variable Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * check replica count by region info Signed-off-by: 5kbpers <[email protected]> * cleanup Signed-off-by: 5kbpers <[email protected]> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <[email protected]> * fix save crcxor Signed-off-by: 5kbpers <[email protected]> * fix decode the key of placement rule Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Update pkg/task/restore.go Co-Authored-By: 3pointer <[email protected]> Co-authored-by: 3pointer <[email protected]> * summary: put summary log at last (#197) * summary: put summary log at last * fix switch sql * *: abstract the progress channel (updateCh) into the glue package (#196) * *: abstract the progress channel (updateCh) into the glue package * restore: fix crash in truncateTS() when the bound is unlimited * task: fix comment Co-authored-by: Ian <[email protected]> * *: refline logs (#189) * tests: disable TLS test (#204) Signed-off-by: Neil Shen <[email protected]> * *: add S3 quick start and few enhancement of log (#202) * README, docker: add quick start about S3 storage Signed-off-by: Neil Shen <[email protected]> * pkg/summary: make sure to output correct summary Signed-off-by: Neil Shen <[email protected]> * cmd, tests: log to terminal if BR_LOG_TO_TERM is set Signed-off-by: Neil Shen <[email protected]> * Update pkg/task/common.go Co-Authored-By: kennytm <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * tests: cat log if br fails Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * restore: add error field to `DownloadResponse` (#195) * restore: add error field to `DownloadResponse` Signed-off-by: 5kbpers <[email protected]> * restore: populate restore cancel error (#207) Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * enhance usability of br (#208) * silenceUsage only when parse cmd flags failed * udpate tidb Co-authored-by: kennytm <[email protected]> * task: do not run checksum if restore failed (#209) * fix incremental bug in llroad test (#199) * restore: filter same table ddl * *: do not return error when backup/restore data is empty * fix create database double during incremental restore * add tests * fix ci * address comment * add skip create sqls (#211) * Revert "tests: disable TLS test (#204)" (#218) This reverts commit e168a60. * doc: add `minio` to dependence list. (#221) The README of test omitted `minio` in the dependence list, which is needed for run the integration test. Co-authored-by: Neil Shen <[email protected]> * move waiting reject stores in import file (#222) * move wait rejectstores into import files * restore: use new table id to search placementRules * Update pkg/restore/import.go Co-Authored-By: Neil Shen <[email protected]> * Update pkg/restore/import.go Co-Authored-By: kennytm <[email protected]> * fix ci Co-authored-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]> * Max index length (#220) * restore: set max-index-length to max * restore:add max-index-length params * address comment * address comment * glue: create schema/table directly with info (#216) * glue: create schema/table directly with info * go.mod: change to use the master version * gluetidb: fix failure to create schema * gluetidb: exclude non-public indices when restoring * go.mod: removed unused replace Co-authored-by: 3pointer <[email protected]> Co-authored-by: Neil Shen <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ian <[email protected]> Co-authored-by: Kolbe Kegel <[email protected]> Co-authored-by: WangXiangUSTC <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: MyonKeminta <[email protected]> Co-authored-by: pingcap-github-bot <[email protected]> Co-authored-by: 庄天翼 <[email protected]> Co-authored-by: 山岚 <[email protected]>

5kbpers added 3 commits March 17, 2020 13:40

restore: remove tiflash replica before restore

5327efc

Signed-off-by: 5kbpers <[email protected]>

Merge branch 'master' into tiflash-restore

abef434

rename errSplit variable

c0e26ca

Signed-off-by: 5kbpers <[email protected]>

3pointer reviewed Mar 17, 2020

View reviewed changes

address comments

8043a9c

Signed-off-by: 5kbpers <[email protected]>

5kbpers force-pushed the tiflash-restore branch from d99b879 to 8043a9c Compare March 17, 2020 06:35

5kbpers added 2 commits March 17, 2020 16:47

check replica count by region info

410c844

Signed-off-by: 5kbpers <[email protected]>

cleanup

3592c89

Signed-off-by: 5kbpers <[email protected]>

3pointer reviewed Mar 17, 2020

View reviewed changes

pkg/restore/client.go Outdated Show resolved Hide resolved

overvenus reviewed Mar 17, 2020

View reviewed changes

3pointer reviewed Mar 17, 2020

View reviewed changes

pkg/restore/client.go Outdated Show resolved Hide resolved

5kbpers added 2 commits March 17, 2020 20:45

save tiflash replica count to backupmeta

901f7b6

Signed-off-by: 5kbpers <[email protected]>

fix save crcxor

a604cf5

Signed-off-by: 5kbpers <[email protected]>

3pointer reviewed Mar 17, 2020

View reviewed changes

pkg/restore/client.go Show resolved Hide resolved

5kbpers added 2 commits March 17, 2020 21:58

fix decode the key of placement rule

44055d1

Signed-off-by: 5kbpers <[email protected]>

address lint

65e47ff

Signed-off-by: 5kbpers <[email protected]>

5kbpers force-pushed the tiflash-restore branch from 65947f4 to 721815e Compare March 17, 2020 14:32

address comments

9a98ffc

Signed-off-by: 5kbpers <[email protected]>

5kbpers force-pushed the tiflash-restore branch from 721815e to 9a98ffc Compare March 17, 2020 14:37

3pointer reviewed Mar 17, 2020

View reviewed changes

pkg/restore/client.go Outdated Show resolved Hide resolved

pkg/restore/client.go Outdated Show resolved Hide resolved

overvenus reviewed Mar 17, 2020

View reviewed changes

close domain after restoring tiflash-replica

63c1d0f

Signed-off-by: 5kbpers <[email protected]>

5kbpers force-pushed the tiflash-restore branch from eb6db46 to 63c1d0f Compare March 18, 2020 03:10

address comments

158eea4

Signed-off-by: 5kbpers <[email protected]>

address comments

5cb4c0e

Signed-off-by: 5kbpers <[email protected]>

3pointer reviewed Mar 18, 2020

View reviewed changes

pkg/task/restore.go Outdated Show resolved Hide resolved

Update pkg/task/restore.go

338c4cf

Co-Authored-By: 3pointer <[email protected]>

overvenus approved these changes Mar 18, 2020

View reviewed changes

3pointer approved these changes Mar 18, 2020

View reviewed changes

3pointer merged commit 6268cde into pingcap:master Mar 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restore: remove tiflash replica before restore #194

restore: remove tiflash replica before restore #194

5kbpers commented Mar 17, 2020

3pointer Mar 17, 2020

5kbpers Mar 17, 2020

overvenus commented Mar 17, 2020

3pointer commented Mar 17, 2020

overvenus commented Mar 17, 2020

overvenus Mar 17, 2020

overvenus Mar 17, 2020

5kbpers Mar 17, 2020

3pointer Mar 17, 2020

3pointer commented Mar 17, 2020

codecov bot commented Mar 17, 2020 •

edited

Loading

overvenus left a comment

overvenus Mar 17, 2020

overvenus Mar 17, 2020

overvenus Mar 17, 2020

5kbpers Mar 18, 2020

overvenus Mar 17, 2020

overvenus Mar 17, 2020

overvenus Mar 17, 2020

3pointer commented Mar 18, 2020

3pointer commented Mar 18, 2020

5kbpers commented Mar 18, 2020

overvenus left a comment

3pointer left a comment

restore: remove tiflash replica before restore #194

restore: remove tiflash replica before restore #194

Conversation

5kbpers commented Mar 17, 2020

What problem does this PR solve?

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

overvenus commented Mar 17, 2020

3pointer commented Mar 17, 2020

overvenus commented Mar 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

3pointer commented Mar 17, 2020

codecov bot commented Mar 17, 2020 • edited Loading

Codecov Report

overvenus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

3pointer commented Mar 18, 2020

3pointer commented Mar 18, 2020

5kbpers commented Mar 18, 2020

overvenus left a comment

Choose a reason for hiding this comment

3pointer left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 17, 2020 •

edited

Loading