Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Engine supports using s3 to store file resources #7114

Closed
2 of 3 tasks
liuzix opened this issue Sep 19, 2022 · 1 comment · Fixed by #7336
Closed
2 of 3 tasks

Engine supports using s3 to store file resources #7114

liuzix opened this issue Sep 19, 2022 · 1 comment · Fixed by #7336
Assignees
@liuzix @CharlesCheung96
Labels
area/engine Issues or PRs related to Dataflow Engine. type/feature Issues about a new feature

Comments

@liuzix
Copy link
Contributor

liuzix commented Sep 19, 2022 •
edited by CharlesCheung96
Loading

@liuzix liuzix added type/feature Issues about a new feature area/engine Issues or PRs related to Dataflow Engine. labels Sep 19, 2022
@liuzix liuzix mentioned this issue Sep 19, 2022
Closed
@CharlesCheung96 CharlesCheung96 mentioned this issue Sep 20, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Sep 27, 2022
@CharlesCheung96
externalresources(engine): refactor local file manager (#7127)
ac46242 
ref #7114
@CharlesCheung96 CharlesCheung96 mentioned this issue Sep 27, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Oct 10, 2022
@CharlesCheung96
externalresources(engine): support s3 as storage (Part 2) (#7204)
4fa7e51 
ref #7114
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Oct 10, 2022
@CharlesCheung96 @3AceShowHand
externalresources(engine): support s3 as storage (Part 2) (pingcap#7204)
42ae387 
ref pingcap#7114
@CharlesCheung96 CharlesCheung96 mentioned this issue Oct 12, 2022
Merged
@CharlesCheung96
Copy link
Contributor

TODO:

  • Figure out the root cause of performance rollback in integration tests
  • Implements remove resource interface for framework
  • Always create a dump task with resource constraints

ti-chi-bot pushed a commit that referenced this issue Oct 17, 2022
@CharlesCheung96
engine(dm, externalresouce): enable s3 for dm (#7336)
2b00b5a 
close #7114, ref #7304
@CharlesCheung96 CharlesCheung96 mentioned this issue Oct 21, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Oct 23, 2022
@CharlesCheung96
engien(dm, externalstorage): generate resource id by storage config (#…
3fb71de 
…7425)

ref #7114
@lance6716 lance6716 mentioned this issue Nov 24, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Nov 24, 2022
@lance6716
lightning(dm): ignore checkpoint error should using ExtStorage (#7702)
507ba3c 
ref #7114
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liuzix liuzix

@CharlesCheung96 CharlesCheung96

Labels
area/engine Issues or PRs related to Dataflow Engine. type/feature Issues about a new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

engine(dm, externalresouce): enable s3 for dm CharlesCheung96/tiflow
2 participants
@liuzix @CharlesCheung96