Incremental Backup & Restore Project

[5kbpers]

Incremental Backup & Restore

Abstract

BR only supports full backup & restore for now, in this project we require to support incremental backup & restore, this is related to BR and TiKV.

Problem Statement

Currently, the procedure of backup is:

1. BR gets a TSO(called BackupTS) from PD
2. BR distributes tasks to TiKVs
3. TiKV scans data with BackupTS, generates SSTs, store them to external storage

And the procedure of restore is:

1. BR resets the TSO of PD with BackupTS(this will be a NOOP, if BackupTS <= current TSO of PD)
2. BR creates the databases and tables, gets the new table schema, constructs key rewrite rules(for rewrite the table id and index id in the keys) with them
3. BR distributes DownloadSST tasks to TiKVs
4. TiKV downloads the given files, rewrites the keys, and save the result file
5. BR distributes IngestSST tasks to TiKVs

Several problems are raised as we support incremental backup & restore:

1. All the data should be scan during the incremental backup is in (last BackupTS, current BackupTS]. Then for more efficient incremental backup , we need a way to skip the unnecessary data.
2. In full restore, There is no old data in the cluster. But in incremental restore, sometimes we need to restore data to a table which already has some records, and they should be covered by the restored data. So we must rewrite the restored data with the TSO from the new cluster, rather reset the TSO with the BackpTS.

Success Criteria

At least support the basic function of incremental backup & restore.

TODO list

• TiKV completes DeltaScanner (500 points / medium)
• TiKV supports backup data in specified TS range (500 points / medium)
• BR supports incremental backup command (500 points / medium)
• TiKV supports to skip SSTs based on MVCC properties (600 points / medium)
• BR and TiKV supports rewrite TS in DownloadSST (600 points / medium)

Score

2700

Mentor(s)

• @5kbpers

Recommended skills

• Golang
• Rust

[Connor1996]

I want to join in this task~

[YangKeao]

I want to join in this task~

[sticnarf]

I want to join in this task~

[youjiali1995]

I want to join in this task~