This repository has been archived by the owner on Jul 24, 2024. It is now read-only.
Merge master to release 3.1 #203
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* restore: fix alter auto increment id for no-primary-key table Signed-off-by: 5kbpers <[email protected]> * add integration test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]>
Signed-off-by: Neil Shen <[email protected]>
* cmd: convert version command to flags Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]>
Signed-off-by: Neil Shen <[email protected]>
* restore-util: Implement split/scatter (#274) * implement split/scatter Signed-off-by: 5kbpers <[email protected]> * init test Signed-off-by: 5kbpers <[email protected]> * redesign output/input of the lib Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * add commments and more tests Signed-off-by: 5kbpers <[email protected]> * add ScanRegions interface to Client Signed-off-by: 5kbpers <[email protected]> * fix potential data race Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * Update pkg/restore-util/client.go Co-Authored-By: kennytm <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * update dependency Signed-off-by: 5kbpers <[email protected]> * resolve conflicts Signed-off-by: 5kbpers <[email protected]> * fix prefix rewrite Signed-off-by: 5kbpers <[email protected]> * add RewriteRule/skip failed scatter region/retry the SplitRegion Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * check if region has peer Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * restore-util: add split retry interval (#277) * reset dependencies to release-3.1 * add split retry interval Signed-off-by: 5kbpers <[email protected]> * fix go.sum Signed-off-by: 5kbpers <[email protected]> * restore-util: wait for scatter region sequentially (#279) * wait for scatter region sequentially Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add on split hook (#281) * restore-util: add on split hook Signed-off-by: Neil Shen <[email protected]> * Nil check onSplit Co-Authored-By: kennytm <[email protected]> * restore-util: fix returned new region is nil (#283) * restore-util: fix returned new region is nil Signed-off-by: 5kbpers <[email protected]> * more logs Signed-off-by: 5kbpers <[email protected]> * *: gofmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * restore-util: call onSplit on splitByRewriteRules (#285) Signed-off-by: Neil Shen <[email protected]> * restore-util: fix overlapped error message (#293) * restore-util: fix overlapped error message Signed-off-by: 5kbpers <[email protected]> * fix log message Signed-off-by: 5kbpers <[email protected]> * reduce error trace Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: log warning when cannot find matched rewrite rule (#299) * restore-util: add method to set placement rules and store labels (#301) * restore-util: add method to set placement rules and store labels Signed-off-by: disksing <[email protected]> * minor fix Signed-off-by: disksing <[email protected]> * address comment Signed-off-by: disksing <[email protected]> * add GetPlacementRules Signed-off-by: disksing <[email protected]> * fix test Signed-off-by: disksing <[email protected]> * restore-util: support batch split (#300) * restore-util: support batch split Signed-off-by: 5kbpers <[email protected]> * go fmt Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * address commits Signed-off-by: 5kbpers <[email protected]> * Update pkg/restore-util/split.go Co-Authored-By: kennytm <[email protected]> * add onSplit callback Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: add upper bound time for waiting for scatter (#305) * restore: fix scatter regions failed Signed-off-by: 5kbpers <[email protected]> * add log Signed-off-by: 5kbpers <[email protected]> * stop waiting for scatter after 3min Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * restore-util: fix wrong url (#306) Signed-off-by: disksing <[email protected]> * restore-util: add warning about unmatched table id (#313) * restore-util: support table partition Signed-off-by: 5kbpers <[email protected]> * fix log Signed-off-by: 5kbpers <[email protected]> * warn table id does not match Signed-off-by: 5kbpers <[email protected]> * add unit tests Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: Neil Shen <[email protected]> * fix compile error Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Ian <[email protected]> Co-authored-by: Neil Shen <[email protected]> * *: prune tidb-tools Signed-off-by: Neil Shen <[email protected]> * restore: address linters suggestions Signed-off-by: Neil Shen <[email protected]> * restore: merge restoreutil into restore Signed-off-by: Neil Shen <[email protected]> * address comment Signed-off-by: Neil Shen <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: disksing <[email protected]> Co-authored-by: Ian <[email protected]>
…tored (#148) * Fixed handling for a dbName that do not exist in the backup being restored * Fixed handling for a dbName that do not exist in the backup being restored
* validate: fix debug meta test ci
* *: extracts runBackup/runRestore in cmd into pkg/task Defines a "Config" structure to store the parsed flags. Use the "black-white-list" structure to define what tables/databases to backup/restore. * go.mod: update tidb to v4.0.0-beta
Co-authored-by: kennytm <[email protected]>
* restore: enhance error handling Signed-off-by: 5kbpers <[email protected]> * unit test Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * fix region epoch error Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * remove `Restore*` Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * add debug log Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Co-Authored-By: kennytm <[email protected]> * Update pkg/restore/import.go Co-Authored-By: kennytm <[email protected]> * fix retry error Signed-off-by: 5kbpers <[email protected]> * handle RegionNotFound error Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]>
* support backup&restore ddl Signed-off-by: 5kbpers <[email protected]> * integration tests Signed-off-by: 5kbpers <[email protected]> * update kvproto Signed-off-by: 5kbpers <[email protected]> * fix integration tests Signed-off-by: 5kbpers <[email protected]> * reduce cyclomatic complexity of `runRestore` Signed-off-by: 5kbpers <[email protected]> * fix test Signed-off-by: 5kbpers <[email protected]> * add unit test Signed-off-by: 5kbpers <[email protected]> * fix tests Signed-off-by: 5kbpers <[email protected]> * disable fast checksum in incremental br Signed-off-by: 5kbpers <[email protected]> * fix no valid key error Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]>
* utils: exclude mock_cluster outside of unit test * utils: remove unused ResultSetToStringSlice() * *: abstract away dependencies of tidb/session into a Glue interface * *: fix hound lint * util,mock: move utils.MockCluster to mock.Cluster * restore: fix test build failure Co-authored-by: 3pointer <[email protected]>
Signed-off-by: Neil Shen <[email protected]>
* *: support tls * move tikv.driver to glue * fix comments
Signed-off-by: Neil Shen <[email protected]>
* backup: add raw backup command
* tests: stable cluster start up Signed-off-by: Neil Shen <[email protected]> * tests: fix unbound var Signed-off-by: Neil Shen <[email protected]> * restore: speed retry on not leader Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * tests: add --cacert flag Signed-off-by: Neil Shen <[email protected]> * make codecov green Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]>
* conn, restore: paginate scan regions Signed-off-by: Neil Shen <[email protected]> * tests: large timeout Signed-off-by: Neil Shen <[email protected]>
* *: unify Range and RangeTree Signed-off-by: Neil Shen <[email protected]> * restore: split restore files into small batch Signed-off-by: Neil Shen <[email protected]> * task: set default restore concurrency to 128 Signed-off-by: Neil Shen <[email protected]> * restore: unused table worker pool Signed-off-by: Neil Shen <[email protected]> * summary: sum up repeated duration and int Signed-off-by: Neil Shen <[email protected]> * rtree: move rtree from utils to pkg Signed-off-by: Neil Shen <[email protected]>
* README, docker: add quick start Signed-off-by: Neil Shen <[email protected]> * cmd: disable some TiDB log Signed-off-by: Neil Shen <[email protected]> * docker: build go-ycsb automatically Signed-off-by: Neil Shen <[email protected]> * cmd: add TODO about TiDB logs Signed-off-by: Neil Shen <[email protected]>
* rtree: move checkFile into backup Signed-off-by: Neil Shen <[email protected]> * *: add license header Signed-off-by: Neil Shen <[email protected]> * Update LICENSE.md Co-Authored-By: kennytm <[email protected]> Co-authored-by: kennytm <[email protected]> Co-authored-by: 3pointer <[email protected]>
…on (#185) Co-authored-by: 3pointer <[email protected]>
* conn: use GetDomain to avoid some TiDB breaking changes Signed-off-by: Neil Shen <[email protected]> * minor usability improvement Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]>
* backup: check safepoint for last backup ts Signed-off-by: 5kbpers <[email protected]> * check lastbackupts > 0 Signed-off-by: 5kbpers <[email protected]> * unhide experimental features Signed-off-by: 5kbpers <[email protected]> * address comment Signed-off-by: 5kbpers <[email protected]> * Update tests/br_z_gc_safepoint/run.sh Co-Authored-By: kennytm <[email protected]> Co-authored-by: kennytm <[email protected]>
* support backupts * address comment * address comment * fix space
Co-authored-by: 3pointer <[email protected]>
Signed-off-by: disksing <[email protected]>
* add grafana scripts * fix Co-authored-by: 3pointer <[email protected]> Co-authored-by: glorv <[email protected]> Co-authored-by: kennytm <[email protected]>
…187) * conn: ignore nodes with label engine=tiflash * conn: disallow TiFlash on restore, only skip TiFlash on backup
* restore: remove tiflash replica before restore Signed-off-by: 5kbpers <[email protected]> * rename errSplit variable Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * check replica count by region info Signed-off-by: 5kbpers <[email protected]> * cleanup Signed-off-by: 5kbpers <[email protected]> * save tiflash replica count to backupmeta Signed-off-by: 5kbpers <[email protected]> * fix save crcxor Signed-off-by: 5kbpers <[email protected]> * fix decode the key of placement rule Signed-off-by: 5kbpers <[email protected]> * address lint Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * close domain after restoring tiflash-replica Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * Update pkg/task/restore.go Co-Authored-By: 3pointer <[email protected]> Co-authored-by: 3pointer <[email protected]>
* summary: put summary log at last * fix switch sql
* *: abstract the progress channel (updateCh) into the glue package * restore: fix crash in truncateTS() when the bound is unlimited * task: fix comment Co-authored-by: Ian <[email protected]>
overvenus
reviewed
Mar 23, 2020
go.mod
Outdated
| github.com/pingcap/parser v3.1.0-beta.1.0.20200317043536-9ebea32e03a6+incompatible | ||
| github.com/pingcap/pd v1.1.0-beta.0.20200214184056-c0bcd8c0f89a // indirect | ||
| github.com/pingcap/parser v3.1.0-beta.1.0.20200318061433-f0b8f6cdca0d+incompatible | ||
| github.com/pingcap/pd/v3 v3.1.0-beta.2.0.20200312100832-1206736bd050 | ||
| github.com/pingcap/pd/v4 v4.0.0-beta.1.0.20200305072537-61d9f9cc35d3 |
|
Could you include f9f6e19? |
Signed-off-by: Neil Shen <[email protected]>
* README, docker: add quick start about S3 storage Signed-off-by: Neil Shen <[email protected]> * pkg/summary: make sure to output correct summary Signed-off-by: Neil Shen <[email protected]> * cmd, tests: log to terminal if BR_LOG_TO_TERM is set Signed-off-by: Neil Shen <[email protected]> * Update pkg/task/common.go Co-Authored-By: kennytm <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * address comments Signed-off-by: Neil Shen <[email protected]> * tests: cat log if br fails Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]>
* restore: add error field to `DownloadResponse` Signed-off-by: 5kbpers <[email protected]>
Signed-off-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]>
* silenceUsage only when parse cmd flags failed * udpate tidb Co-authored-by: kennytm <[email protected]>
* restore: filter same table ddl * *: do not return error when backup/restore data is empty * fix create database double during incremental restore * add tests * fix ci * address comment
|
/run-integration-tests |
1 similar comment
|
/run-integration-tests |
The README of test omitted `minio` in the dependence list, which is needed for run the integration test. Co-authored-by: Neil Shen <[email protected]>
* move wait rejectstores into import files * restore: use new table id to search placementRules * Update pkg/restore/import.go Co-Authored-By: Neil Shen <[email protected]> * Update pkg/restore/import.go Co-Authored-By: kennytm <[email protected]> * fix ci Co-authored-by: Neil Shen <[email protected]> Co-authored-by: kennytm <[email protected]>
* restore: set max-index-length to max * restore:add max-index-length params * address comment * address comment
* glue: create schema/table directly with info * go.mod: change to use the master version * gluetidb: fix failure to create schema * gluetidb: exclude non-public indices when restoring * go.mod: removed unused replace Co-authored-by: 3pointer <[email protected]>
overvenus
approved these changes
Apr 2, 2020
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
What is changed and how it works?
Check List
Tests
Code changes
Side effects
Related changes