Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support tiflash backup and restore during volume snapshot #4812

Merged
merged 8 commits into from
Jan 8, 2023

Conversation

fengou1
Copy link
Contributor

@fengou1 fengou1 commented Dec 28, 2022

What problem does this PR solve?

add tiflash support for volume-snapshot backup and restore

Closes #4807

What is changed and how does it work?

  1. after BR 1st restore done, start TiKV
  2. BR 2nd restore will set pd allocate id and after that, the 2nd restore unmask the pd recovering mark.
  3. TiFlash detects if PD recovering mark is unset, start TiFlash
  4. wait BR 2nd restore done

Code changes

  • Has Go code change
  • Has CI related scripts change

Tests

  • Unit test
  • E2E test
  • Manual test
  • backup without tiflash - PASS
  • restore without tiflash - PASS
  • backup without tiflash - PASS
  • restore with tiflash - PASS
  • backup with tiflash - PASS
  • restore with tiflash - PASS
  • backup with tiflash - PASS
  • restore without tif lash - PASS
  • No code

Side effects

  • Breaking backward compatibility
  • Other side effects:

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.


@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 28, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • WangLe1321
  • WizardXiao

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@codecov-commenter
Copy link

codecov-commenter commented Dec 28, 2022

Codecov Report

Merging #4812 (b51f886) into master (e875017) will increase coverage by 8.29%.
The diff coverage is 26.72%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4812      +/-   ##
==========================================
+ Coverage   60.10%   68.40%   +8.29%     
==========================================
  Files         224      228       +4     
  Lines       25261    28418    +3157     
==========================================
+ Hits        15182    19438    +4256     
+ Misses       8650     7523    -1127     
- Partials     1429     1457      +28     
Flag Coverage Δ
e2e 52.70% <2.88%> (?)
unittest 60.00% <30.61%> (-0.10%) ⬇️

@fengou1
Copy link
Contributor Author

fengou1 commented Dec 29, 2022

/test pull-e2e-kind-br

@fengou1
Copy link
Contributor Author

fengou1 commented Dec 29, 2022

/test pull-e2e-kind-across-kubernetes

pkg/backup/restore/restore_manager.go Outdated Show resolved Hide resolved
pkg/pdapi/fake_pdapi.go Outdated Show resolved Hide resolved
@fengou1
Copy link
Contributor Author

fengou1 commented Jan 3, 2023

/cc @WangLe1321 @WizardXiao

Copy link
Contributor

@WangLe1321 WangLe1321 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 7, 2023

/test pull-e2e-kind-br

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 7, 2023

/test pull-e2e-kind

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 8, 2023

/test pull-e2e-kind

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 8, 2023

/test pull

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 8, 2023

/test pull-e2e-kind-serial

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 8, 2023

/test pull-e2e-kind

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 8, 2023

/test pull-e2e-kind-br

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 8, 2023

/test pull-e2e-kind-across-kubernetes

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 8, 2023

/test pull-e2e-kind-tikv-scale-simultaneously

@fengou1
Copy link
Contributor Author

fengou1 commented Jan 8, 2023

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: b51f886

@fengou1 fengou1 self-assigned this Jan 8, 2023
@ti-chi-bot ti-chi-bot merged commit 61fcf32 into pingcap:master Jan 8, 2023
WizardXiao pushed a commit to WizardXiao/tidb-operator that referenced this pull request Mar 10, 2023
WizardXiao added a commit that referenced this pull request Mar 11, 2023
* feat: support tiflash backup and restore during volume snapshot (#4812)

* feat: calc the backup size from snapshot storage usage (#4819)

* fix backup failed when pod was auto restarted by k8s (#4883)

* init code for test

* just clean before backup data

* delete test code

* import pingcap/errors

* add check version

* remove test code

* add running status check

* add restart condition to clarify logic

* fix status update

* fix ut

* br: ensure pvc names sequential for ebs restore (#4888)

* BR: Restart backup when backup job/pod unexpected failed by k8s (#4895)

* init code for test

* just clean before backup data

* delete test code

* import pingcap/errors

* add check version

* remove test code

* add running status check

* add restart condition to clarify logic

* fix status update

* fix ut

* init code

* update crd reference

* fix miss update retry count

* add retry limit as constant

* init runnable code

* refine main controller logic

* add some note

* address some comments

* init e2e test code

* add e2e env to extend backup time

* add e2e env for test

* fix complie

* just test kill pod

* refine logic

* use pkill to kill pod

* fix reconcile

* add kill pod log

* add more log

* add more log

* try kill pod only

* wait and kill running backup pod

* add wait for pod failed

* fix wait pod running

* use killall backup to kill pod

* use pkill -9 backup

* kill pod until pod is failed

* add ps to debug

* connect commands by semicolon

* kill pod by signal 15

* use panic simulate kill pod

* test all kill pod test

* remove useless log

* add original reason of job or pod failure

* rename BackupRetryFailed to BackupRetryTheFailed

* BR: Auto truncate log backup in backup schedule (#4904)

* init schedule log backup code

* add run log backup code

* update api

* refine some nodes

* refine cacluate logic

* add ut

* fix make check

* add log backup test

* refine code

* fix notes

* refine function names

* fix conflict

* fix: add a new check for encryption during the volume snapshot restore (#4914)

* br: volume-snapshot may lead to a panic when there is no block change between two snapshot (#4922)

* br: refine BackoffRetryPolicy time format (#4925)

* refine BackoffRetryPolicy time format

* fix some ut

---------

Co-authored-by: fengou1 <[email protected]>
Co-authored-by: WangLe1321 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TiKV pods can't be created when execute EBS snapshot restore with Tiflash
5 participants