ebs br: provide fsr warmup to tikv data volumes #47272

BornChanger · 2023-09-25T18:51:32Z

What problem does this PR solve?

Issue Number:
close #47271
close #47273

Problem Summary:

What is changed and how it works?

This PR addresses 2 issues

EBS snapshot restore can do FSR based warmup to tikv data volumes. A new parameter ofuse-fsr is introduced to br command. Set it to true to use FSR to warmup tikv data volumes.
Get rid of unnecessary removal of some users from mysql.user table at the end of restore

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
Run real EBS restore job and check the BR pod log to verify.
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2023-09-25T18:51:42Z

Hi @BornChanger. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tiprow · 2023-09-25T18:51:54Z

Hi @BornChanger. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-09-25T18:52:24Z

/ok-to-test

ti-chi-bot · 2023-09-25T18:52:39Z

@BornChanger: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tiprow · 2023-09-25T18:52:53Z

@BornChanger: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-09-25T18:53:07Z

/cc @WangLe1321 @YuJuncen

ti-chi-bot · 2023-09-25T18:53:09Z

@BornChanger: GitHub didn't allow me to request PR reviews from the following users: WangLe1321.

Note that only pingcap members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @WangLe1321 @YuJuncen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-09-26T01:26:32Z

/component br

br/pkg/aws/ebs.go

br/pkg/task/restore_ebs_meta.go

codecov · 2023-10-11T09:17:23Z

Codecov Report

Merging #47272 (613654e) into master (ca977c5) will increase coverage by 1.1894%.
Report is 4 commits behind head on master.
The diff coverage is 0.0000%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #47272        +/-   ##
================================================
+ Coverage   71.6203%   72.8097%   +1.1894%     
================================================
  Files          1401       1424        +23     
  Lines        405776     412310      +6534     
================================================
+ Hits         290618     300202      +9584     
+ Misses        95371      93239      -2132     
+ Partials      19787      18869       -918

Flag	Coverage Δ
integration	`42.1627% <0.0000%> (?)`
unit	`71.6441% <0.0000%> (+0.0238%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`54.0503% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`48.6770% <0.0000%> (-4.3375%)`	⬇️

br/pkg/aws/ebs.go

ti-chi-bot · 2023-10-17T04:05:41Z

@WangLe1321: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-10-18T08:31:20Z

/retest

tiprow · 2023-10-18T08:31:52Z

@BornChanger: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-10-18T13:38:09Z

/test pull-br-integration-test

tiprow · 2023-10-18T13:38:44Z

@BornChanger: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-br-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: BornChanger <[email protected]>

ti-chi-bot · 2023-10-27T11:59:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, WangLe1321, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~br/OWNERS~~ [3pointer,YuJuncen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-10-27T11:59:23Z

[LGTM Timeline notifier]

Timeline:

2023-10-27 02:01:12.013714921 +0000 UTC m=+2572869.600825060: ☑️ agreed by 3pointer.
2023-10-27 11:59:22.068764387 +0000 UTC m=+2608759.655874532: ☑️ agreed by YuJuncen.

BornChanger · 2023-10-27T12:01:59Z

/label may-affects-6.5

BornChanger · 2023-10-27T12:02:32Z

/label may-affects-7.1 may-affects-7.5

BornChanger · 2023-10-27T12:20:22Z

/retest

BornChanger · 2023-10-27T12:24:29Z

/test tiprow_fast_test

ti-chi-bot · 2023-10-27T12:24:32Z

@BornChanger: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build
/test check-dev
/test check-dev2
/test mysql-test
/test unit-test

The following commands are available to trigger optional jobs:

/test canary-notify-when-compatibility-sections-changed
/test pingcap/tidb/canary_ghpr_unit_test
/test pull-br-integration-test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-ddl-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-mysql-client-test
/test pull-sqllogic-test
/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test

In response to this:

/test tiprow_fast_test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-10-27T13:15:28Z

/label needs-cherry-pick-release-6.5

ti-chi-bot · 2023-10-27T13:16:15Z

In response to a cherrypick label: new pull request created to branch release-6.5: #48051.

BornChanger · 2023-10-27T13:16:43Z

/label needs-cherry-pick-release-7.1

BornChanger · 2023-10-27T13:16:49Z

/label needs-cherry-pick-release-7.5

ti-chi-bot · 2023-10-27T13:17:26Z

In response to a cherrypick label: new pull request created to branch release-7.1: #48052.

ti-chi-bot · 2023-10-27T13:17:31Z

In response to a cherrypick label: new pull request created to branch release-7.5: #48053.

…cap#48051) (pingcap#21) Co-authored-by: Ti Chi Robot <[email protected]>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 25, 2023

ti-chi-bot bot requested a review from YuJuncen September 25, 2023 18:53

BornChanger force-pushed the support_esb_fsr branch 2 times, most recently from c92c973 to 5c13221 Compare September 26, 2023 00:33

ti-chi-bot bot added the component/br This issue is related to BR of TiDB. label Sep 26, 2023

WangLe1321 reviewed Sep 26, 2023

View reviewed changes

br/pkg/aws/ebs.go Outdated Show resolved Hide resolved

br/pkg/task/restore_ebs_meta.go Show resolved Hide resolved

BornChanger force-pushed the support_esb_fsr branch from 5c13221 to a386595 Compare October 11, 2023 08:57

BornChanger requested a review from WangLe1321 October 11, 2023 09:14

BornChanger force-pushed the support_esb_fsr branch from 621357e to cd1bb95 Compare October 11, 2023 10:27

WangLe1321 reviewed Oct 12, 2023

View reviewed changes

br/pkg/aws/ebs.go Outdated Show resolved Hide resolved

br/pkg/aws/ebs.go Outdated Show resolved Hide resolved

br/pkg/aws/ebs.go Show resolved Hide resolved

BornChanger mentioned this pull request Oct 16, 2023

EBS BR support to multiple k8s cluster TiDB pingcap/tidb-operator#5003

Open

BornChanger force-pushed the support_esb_fsr branch from 7dd5f8a to 5d37fac Compare October 16, 2023 16:53

WangLe1321 approved these changes Oct 17, 2023

View reviewed changes

BornChanger force-pushed the support_esb_fsr branch from 42cee43 to 8596138 Compare October 23, 2023 14:15

*: address comments for 2nd round

613654e

Signed-off-by: BornChanger <[email protected]>

BornChanger force-pushed the support_esb_fsr branch from ff128cd to 613654e Compare October 27, 2023 11:57

YuJuncen approved these changes Oct 27, 2023

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Oct 27, 2023

ti-chi-bot bot added the may-affects-6.5 label Oct 27, 2023

ti-chi-bot bot merged commit 30288c7 into pingcap:master Oct 27, 2023
16 of 24 checks passed

ti-chi-bot bot added the needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. label Oct 27, 2023

ti-chi-bot mentioned this pull request Oct 27, 2023

ebs br: provide fsr warmup to tikv data volumes (#47272) #48051

Merged

13 tasks

ti-chi-bot bot added the needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. label Oct 27, 2023

ti-chi-bot bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Oct 27, 2023

This was referenced Oct 27, 2023

ebs br: provide fsr warmup to tikv data volumes (#47272) #48052

Merged

ebs br: provide fsr warmup to tikv data volumes (#47272) #48053

Merged

ti-chi-bot bot pushed a commit that referenced this pull request Oct 27, 2023

ebs br: provide fsr warmup to tikv data volumes (#47272) (#48051)

a211b67

ti-chi-bot bot pushed a commit that referenced this pull request Dec 8, 2023

ebs br: provide fsr warmup to tikv data volumes (#47272) (#48052)

6b72ccd

ti-chi-bot bot pushed a commit that referenced this pull request Feb 7, 2024

ebs br: provide fsr warmup to tikv data volumes (#47272) (#48053)

b2d3fb9

guoshouyan pushed a commit to guoshouyan/tidb that referenced this pull request Mar 5, 2024

ebs br: provide fsr warmup to tikv data volumes (pingcap#47272) (ping…

7389440

…cap#48051) (pingcap#21) Co-authored-by: Ti Chi Robot <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ebs br: provide fsr warmup to tikv data volumes #47272

ebs br: provide fsr warmup to tikv data volumes #47272

BornChanger commented Sep 25, 2023 •

edited

Loading

ti-chi-bot bot commented Sep 25, 2023

tiprow bot commented Sep 25, 2023

BornChanger commented Sep 25, 2023

ti-chi-bot bot commented Sep 25, 2023

tiprow bot commented Sep 25, 2023

BornChanger commented Sep 25, 2023

ti-chi-bot bot commented Sep 25, 2023

BornChanger commented Sep 26, 2023

codecov bot commented Oct 11, 2023 •

edited

Loading

ti-chi-bot bot commented Oct 17, 2023

BornChanger commented Oct 18, 2023

tiprow bot commented Oct 18, 2023

BornChanger commented Oct 18, 2023

tiprow bot commented Oct 18, 2023

ti-chi-bot bot commented Oct 27, 2023

ti-chi-bot bot commented Oct 27, 2023

BornChanger commented Oct 27, 2023

BornChanger commented Oct 27, 2023

BornChanger commented Oct 27, 2023

BornChanger commented Oct 27, 2023

ti-chi-bot bot commented Oct 27, 2023

BornChanger commented Oct 27, 2023

ti-chi-bot commented Oct 27, 2023

BornChanger commented Oct 27, 2023

BornChanger commented Oct 27, 2023

ti-chi-bot commented Oct 27, 2023

ti-chi-bot commented Oct 27, 2023

ebs br: provide fsr warmup to tikv data volumes #47272

ebs br: provide fsr warmup to tikv data volumes #47272

Conversation

BornChanger commented Sep 25, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot bot commented Sep 25, 2023

tiprow bot commented Sep 25, 2023

BornChanger commented Sep 25, 2023

ti-chi-bot bot commented Sep 25, 2023

tiprow bot commented Sep 25, 2023

BornChanger commented Sep 25, 2023

ti-chi-bot bot commented Sep 25, 2023

BornChanger commented Sep 26, 2023

codecov bot commented Oct 11, 2023 • edited Loading

Codecov Report

ti-chi-bot bot commented Oct 17, 2023

BornChanger commented Oct 18, 2023

tiprow bot commented Oct 18, 2023

BornChanger commented Oct 18, 2023

tiprow bot commented Oct 18, 2023

ti-chi-bot bot commented Oct 27, 2023

ti-chi-bot bot commented Oct 27, 2023

[LGTM Timeline notifier]

BornChanger commented Oct 27, 2023

BornChanger commented Oct 27, 2023

BornChanger commented Oct 27, 2023

BornChanger commented Oct 27, 2023

ti-chi-bot bot commented Oct 27, 2023

BornChanger commented Oct 27, 2023

ti-chi-bot commented Oct 27, 2023

BornChanger commented Oct 27, 2023

BornChanger commented Oct 27, 2023

ti-chi-bot commented Oct 27, 2023

ti-chi-bot commented Oct 27, 2023

BornChanger commented Sep 25, 2023 •

edited

Loading

codecov bot commented Oct 11, 2023 •

edited

Loading