From 4b47114ee9b53d0f1dced3e8ff63c9e739821b4a Mon Sep 17 00:00:00 2001 From: Leavrth Date: Thu, 1 Dec 2022 14:33:38 +0800 Subject: [PATCH 1/3] draft Signed-off-by: Leavrth --- .../2022-10-26-checkpoint-for-backup.md | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 docs/design/2022-10-26-checkpoint-for-backup.md diff --git a/docs/design/2022-10-26-checkpoint-for-backup.md b/docs/design/2022-10-26-checkpoint-for-backup.md new file mode 100644 index 0000000000000..bf951f66967a3 --- /dev/null +++ b/docs/design/2022-10-26-checkpoint-for-backup.md @@ -0,0 +1,26 @@ +# Proposal: Checkpoint For Backup +- Author(s): [Leavrth](https://github.com/Leavrth) +- Tracking Issues: https://github.com/pingcap/tidb/issues/38647 + +## Abstract + +This proposal aims to support `Checkpoint For Backup` and describe what `Checkpoint For Backup` should look like and how to implement it. + +## Background + +Snapshot backup might be interrupted due to recoverable errors, such as disk exhaustion, node crash and network timeout. After these errors are addressed, users can try the next backup. However, data that is backed up before the interruption would be invalidated, and everything needs to start from scratch. For large-scale clusters, this incurs considerable extra cost. + +## Detailed Design + +### Implementation Overview + +At the begining of a snapshot backup, BR encodes the table by table's ID into a `range`. All the data's keys of the table are involved of the `range`. + +Then BR sends a backup request with the `range` to all the TiKV nodes. Once each TiKV node receives the request, it begins to back up the regions with leader status involved of the `range`. And once each region finishes being backed up, the TiKV node, which the region belongs to, sends a response with the overlapping range of `region` and `range`. When BR receives the response, the data of the overlapping range is regarded as having backed up. + +The response contains not only the overlappping range, but also the paths of the backup files. BR persist these meta-information to the external storage. + +After BR exits due to an error and the error is addressed, users try to run BR again. At the begining of this snapshot backup, + + + From 2de6717ed238f58ac4309f99fddb96eb59790e59 Mon Sep 17 00:00:00 2001 From: Leavrth Date: Thu, 1 Dec 2022 21:18:18 +0800 Subject: [PATCH 2/3] draft Signed-off-by: Leavrth --- .../2022-10-26-checkpoint-for-backup.md | 24 +++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/docs/design/2022-10-26-checkpoint-for-backup.md b/docs/design/2022-10-26-checkpoint-for-backup.md index bf951f66967a3..0445d484b76f3 100644 --- a/docs/design/2022-10-26-checkpoint-for-backup.md +++ b/docs/design/2022-10-26-checkpoint-for-backup.md @@ -18,9 +18,29 @@ At the begining of a snapshot backup, BR encodes the table by table's ID into a Then BR sends a backup request with the `range` to all the TiKV nodes. Once each TiKV node receives the request, it begins to back up the regions with leader status involved of the `range`. And once each region finishes being backed up, the TiKV node, which the region belongs to, sends a response with the overlapping range of `region` and `range`. When BR receives the response, the data of the overlapping range is regarded as having backed up. -The response contains not only the overlappping range, but also the paths of the backup files. BR persist these meta-information to the external storage. +The response contains not only the overlappping range, but also the paths of the backup files. BR persist these `meta-information` to the external storage. -After BR exits due to an error and the error is addressed, users try to run BR again. At the begining of this snapshot backup, +After BR exits due to an error and the error is addressed, users try to run BR again. At the begining of this snapshot backup, BR encodes the table by table's ID into a `range`, which is called `the origin range` now, like before. Then BR loads the backed up data's `meta-informationt`, which contains these backed up data's ranges called `sub ranges` now. And finally the difference ranges between `the origin range` and `sub ranges` are the data that needs to be backed up this time. +Instead of sending the request with `the origin range`, BR sends the request with the difference ranges to all the TiKV nodes. To avoid an increase in the number of requests, we add a new field for the backup request. The protobuf related change shown below: +```protobuf +message BackupRequest { + uint64 cluster_id = 1; + bytes start_key = 2; + bytes end_key = 3; + + // ... + + // with checkpoint, some subintervals of the range have been backed up and recorded. + // only the remaining sub ranges of the range need to be backed up this time. + repeated kvrpcpb.KeyRange sub_ranges = 16; +} +``` + +### Limitations and future Work + +Once BR runs snapshot backup, it tries to write a lock file to the external storage at first. This prevents different BRs from performing the same task. + +However, BR skips checking the lock file when checkpoint metadata exists in the external storage. So there may be different BRs performing the same task at the same time. From 2e29b549ae7056491b12f6d2310ccdabf9825916 Mon Sep 17 00:00:00 2001 From: Leavrth Date: Fri, 10 Feb 2023 22:07:11 +0800 Subject: [PATCH 3/3] test integration test Signed-off-by: Leavrth --- br/tests/br_views_and_sequences/run.sh | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/br/tests/br_views_and_sequences/run.sh b/br/tests/br_views_and_sequences/run.sh index 71403d0a0f6a3..491a73f77c216 100755 --- a/br/tests/br_views_and_sequences/run.sh +++ b/br/tests/br_views_and_sequences/run.sh @@ -56,6 +56,16 @@ views_count=$(run_sql "select count(*) c, sum(m) s from $DB.view_3;" | tail -2 | run_sql "insert into $DB.table_2 (c) values (33);" seq_val=$(run_sql "select a >= 8 and b >= 4 as g from $DB.table_2 where c = 33;" | tail -1) +val=$(run_sql "SELECT LASTVAL($DB.seq_1);") +echo "$val" +val=$(run_sql "SELECT LASTVAL($DB.seq_2);") +echo "$val" +val=$(run_sql "SELECT NEXTVAL($DB.seq_1);") +echo "$val" +val=$(run_sql "SELECT NEXTVAL($DB.seq_2);") +echo "$val" +val=$(run_sql "SELECT * from $DB.table_2;") +echo "$val" [ "$seq_val" = 'g: 1' ] run_sql "insert into $DB.auto_inc values ();"