backup: debounce fine-grained backup #1036
Conversation
pkg/backup/client.go
Outdated
| // If no progress, backoff 10s for debouncing. | ||
| // 10s is the default interval of stores sending a heartbeat to the PD. | ||
| // And is the average new leader election timeout, which would be a reasonable back off time. | ||
| if !hasProgress && backoffMill < 10000 { |
There was a problem hiding this comment.
if !hasProgress is enough?
|
/lgtm |
|
/lgtm |
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by writing |
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: faffdbc
|
Signed-off-by: ti-srebot <[email protected]>
|
cherry pick to release-5.0 in PR #1037 |
Signed-off-by: ti-srebot <[email protected]>
|
cherry pick to release-4.0 in PR #1067 |
What problem does this PR solve?
Sometimes, TiKV make empty response stream from backing up request. (This seems buggy, but not sure.)
In this condition, fine-grained backup would crazily send
backupRPC, if there are enough resource, thing wouldn't be so bad: when PD make progress, real leader can be found, and everything should back to normal, but for our CI, the busiest node of our community, un, ya see.Note the timestamp. There are about 100k+ or even more logs like this.
What is changed and how it works?
Now, if fine-grained backup make no progress, it would backoff for 10 seconds.
The backoff time is an experience value, by analyzing the local log. Let's change it if necessary.
Check List
Tests
Release Note