Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix gateway server bug and interrupt handling for chunk dispatching #868

Merged
merged 221 commits into from
Jun 15, 2023
Merged

Fix gateway server bug and interrupt handling for chunk dispatching #868

merged 221 commits into from
Jun 15, 2023

Conversation

sarahwooders
Copy link
Contributor

@sarahwooders sarahwooders commented Jun 14, 2023 •
edited
Loading

This fixes two bugs:

  • During chunk dispatching for CLI transfers, Ctrl-C could not be used to exit out of the transfer due to threads not being properly filled. This is fixed by adding exit signals to the tracker and stopped the multipart threads when the tracker is exited.
  • Previous self.auth in the Server instance was modified to contain credentials for all clouds that are initialized, however this was not properly updated for child classes.

lynnliu030 and others added 30 commits November 29, 2022 13:36
@lynnliu030 @parasj @sarahwooders
Add Broadcast client API interface (skyplane-project#675)
e66f065 
This change introduces an API for Skyplane Broadcast

Todos:
- [x] Fix provisioning in BroadcastDataplane
  - Reuse provision loop via inheritance
  - Move `_start_gateway` to a class method and override it
  - Adapt broadcast to use `bound_nodes`
- [x] Add BroadcastCopyJob (ideally extend CopyJob)
- [x] Update tracker to monitor broadcast jobs
- [x] Add multipart support 
- [x] Fix dependency issue via adding dockerfile and bc_requirements 
- [x] Integrate with gateway and test the monitoring side 

Co-authored-by: Paras Jain <[email protected]>
Co-authored-by: Sarah Wooders <[email protected]>
@sarahwooders
add bucket replication script for experiments
8b842c1
@sarahwooders
fix region logic
d6a5a9a
@sarahwooders
merge
e15d8fb
@sarahwooders
add log copying
dfecc9c
@lynnliu030
update transfer cost grid
5881d45
@sarahwooders
switch batch to recursive
9f9c4dd
@sarahwooders
Merge branch 'broadcast' of github.com:skyplane-project/skyplane into…
771f4dd 
… broadcast
@lynnliu030
change the upload ids
43a9514
@lynnliu030
put
0133f32
@lynnliu030
changes for broadcast multipart
0da6d43
@sarahwooders
Mux_and fix (skyplane-project#718)
59b4176
@sarahwooders
couple gateawy fixes
99f584d
@sarahwooders
just return 0
d785084
@lynnliu030
fix bc verification
d6f7065
@lynnliu030
print per-dst remaining bytes
1093d5f
@sarahwooders
fix obj store wait time
8fb9a72
@sarahwooders
Merge branch 'multipart' of github.com:skyplane-project/skyplane into…
689386b 
… multipart
@sarahwooders
assert completed = file deleted
9c56692
@sarahwooders
minor fix
b5350d0
@sarahwooders
accidentally deleted line
b4eb68d
@sarahwooders
fix issue with terminal operators
28ca507
@lynnliu030
update tput profile & ILP
c7e56da
@lynnliu030
update run config
651fb97
@sarahwooders
reduce client parallism
3590de7
@sarahwooders
count num processes
aa83a89
@lynnliu030
add gw programs
fb460b6
@sarahwooders
add back print
bbbfd30
@sarahwooders
modify gateway program processing on gateway side
207b8f4
@sarahwooders
working 5 dest transfer
5a798f7
@sarahwooders sarahwooders changed the title Fix gateway server bug Fix gateway server bug and interrupt handling for chunk dispatching Jun 14, 2023
@sarahwooders sarahwooders merged commit 2696494 into skyplane-project:main Jun 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abiswal2001 abiswal2001 Awaiting requested review from abiswal2001

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sarahwooders @lynnliu030 @parasj @abiswal2001