Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

engine(dm, externalresouce): enable s3 for dm #7336

Merged
merged 3 commits into from
Oct 17, 2022
Merged

engine(dm, externalresouce): enable s3 for dm #7336

merged 3 commits into from
Oct 17, 2022

Conversation

CharlesCheung96
Copy link
Contributor

@CharlesCheung96 CharlesCheung96 commented Oct 12, 2022 •
edited by lance6716
Loading

What problem does this PR solve?

Issue Number: close #7114, ref #7304

What is changed and how it works?

  1. s3 resources are used by default in dm tasks.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

`None`.

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Oct 12, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. area/engine Issues or PRs related to Dataflow Engine. labels Oct 12, 2022
@GMHDBJD
Copy link
Contributor

GMHDBJD commented Oct 12, 2022

can you also deploy minio in helm? https://github.com/pingcap/tiflow/blob/master/deployments/engine/helm/tiflow/values.yaml

@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. do-not-merge/needs-linked-issue size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 12, 2022
@CharlesCheung96
Copy link
Contributor Author

can you also deploy minio in helm? https://github.com/pingcap/tiflow/blob/master/deployments/engine/helm/tiflow/values.yaml

We do not need to deploy minio directly in helm, since a public minio service is available on test_infra. Here we provide a configuration file to deploy minio in k8s, which can be deployed manually in githuc action.

@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

GMHDBJD
GMHDBJD reviewed Oct 13, 2022
View reviewed changes
engine/jobmaster/dm/utils.go Outdated Show resolved Hide resolved
engine/jobmaster/dm/utils.go Show resolved Hide resolved
@CharlesCheung96
Copy link
Contributor Author

chao tests verified in https://github.com/CharlesCheung96/tiflow/actions/runs/3247512899

@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@ti-chi-bot ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Oct 17, 2022
lance6716
lance6716 reviewed Oct 17, 2022
View reviewed changes
Copy link
Contributor

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm

deployments/engine/docker-compose/3m3e_with_s3.yaml Show resolved Hide resolved
deployments/engine/docker-compose/config/executor_with_s3.toml Show resolved Hide resolved
deployments/engine/docker-compose/config/master_with_s3.toml Show resolved Hide resolved
engine/pkg/externalresource/model/model.go Outdated Show resolved Hide resolved
engine/pkg/externalresource/model/model.go Outdated Show resolved Hide resolved
@lance6716
Copy link
Contributor

Can some TODO in chaos test also be fixed? @GMHDBJD

@CharlesCheung96 CharlesCheung96 mentioned this pull request Oct 17, 2022
Closed
@CharlesCheung96
Copy link
Contributor Author

/run-verify
/run-engine-integration-test

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 17, 2022
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 17, 2022
@amyangfei
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: efcac30

@ti-chi-bot ti-chi-bot added status/can-merge Indicates a PR has been approved by a committer. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 17, 2022
@ti-chi-bot ti-chi-bot removed status/can-merge Indicates a PR has been approved by a committer. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 17, 2022
@CharlesCheung96
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 22a04e9

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Oct 17, 2022
@amyangfei
Copy link
Contributor

/run-engine-integration-test

@ti-chi-bot ti-chi-bot merged commit 2b00b5a into pingcap:master Oct 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyangfei amyangfei amyangfei approved these changes

@lance6716 lance6716 lance6716 approved these changes

@GMHDBJD GMHDBJD Awaiting requested review from GMHDBJD

Assignees
No one assigned
Labels
area/engine Issues or PRs related to Dataflow Engine. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Engine supports using s3 to store file resources
5 participants
@CharlesCheung96 @ti-chi-bot @GMHDBJD @lance6716 @amyangfei