Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lightning: support table-level local-backend #35148

Closed
sleepymole opened this issue Jun 5, 2022 · 0 comments · Fixed by #34130
Closed

lightning: support table-level local-backend #35148

sleepymole opened this issue Jun 5, 2022 · 0 comments · Fixed by #34130
Assignees
Labels
component/lightning This issue is related to Lightning of TiDB. type/enhancement The issue or PR belongs to an enhancement.

Comments

@sleepymole
Copy link
Contributor

sleepymole commented Jun 5, 2022

Enhancement

We want to use tidb-lightning to import data to a cluster that is running online services. To reduce the impact on online services, we need to reduce the impact of local-backend to the table-level.

We have to do several enhancements:

First, tidb-lightning shouldn't disable the whole PD scheduler during importing. Instead, lightning should only disable the scheduler on the region that is being imported.tikv/pd#4695 supported TTL for region label. We can use the region label to disable the scheduler on a specific key range. With the help of TTL, we don't need to worry that the scheduler won't be recovered when tidb-lightning is crashed. For region label, please see tikv/pd#3839 for more details.

Second, tidb-lightning need to provide a mechanism to limit the write speed to each TiKV store. We can add a new config tikv-importer.store-write-bwlimit and limit the write speed to TiKV nodes.

@sleepymole sleepymole added type/enhancement The issue or PR belongs to an enhancement. component/lightning This issue is related to Lightning of TiDB. labels Jun 5, 2022
@sleepymole sleepymole self-assigned this Jun 5, 2022
@sleepymole sleepymole changed the title lightning: support disable scheduler by key range lightning: support table-level local-backend Jul 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/lightning This issue is related to Lightning of TiDB. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant