Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore: resume_from_checkpoint #38143

Merged
merged 113 commits into from
Aug 9, 2023
Merged

[air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore: resume_from_checkpoint #38143

merged 113 commits into from
Aug 9, 2023

Conversation

justinvyu
Copy link
Contributor

@justinvyu justinvyu commented Aug 5, 2023 •
edited
Loading

Why are these changes needed?

This PR supports Trainer(resume_from_checkpoint) with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@justinvyu
Pipe storage context to Trainable (used now for Trainable syncing)
abb1307 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Don't use the storage context in the trial/trainable
f6ff90a 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Disable all trainable syncing in new codepath
562369f 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Pipe storage context to Train workers (not actually used yet)
95a3d20 
Signed-off-by: Justin Yu <[email protected]>

Update to use the new checkpoint id attribute

Signed-off-by: Justin Yu <[email protected]>

Add todo comment to remove legacy path

Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Fix race condition for setting checkpoint_uri
484e67f 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Fix cyclical import
2148669 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Add simple trainer test
8c856b8 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Add legacy prefix to train session checkpoint uri
78c525f 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Add new checkpoint class
e97f471 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
New train session report implementation using new checkpoint
64945be 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Simplify checkpoint propagation from user code (in worker) -> trainer…
c6480c9 
… -> driver

Signed-off-by: Justin Yu <[email protected]>
@justinvyu
New tune session.report
c681ccb 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Save direction works with new checkpoint API
795bafe 
Signed-off-by: Justin Yu <[email protected]>

Fix lint

Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Update test with e2e trainer test
8a084bc 
Signed-off-by: Justin Yu <[email protected]>

Fix lint

Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Make callback supporting new checkpoint a todo for now
725d802 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Remove unnecessary comment
877acb9 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into air/…
ee4ccbd 
…persistence/storage_context_to_worker_temp
@justinvyu
Separate out the new set checkpoint id from the old set checkpoint uri
88042b3 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into air/…
a5eeab2 
…persistence/new_checkpoint

Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Update id -> index
a6cd9dc 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Address comments on error to raise with old ckpt type
01f34bb 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Move checkpoint upload logic to a helper fn of storage ctx
65e7a27 
Signed-off-by: Justin Yu <[email protected]>

Fix lint for session.py

Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Drop a checkpoint marker after uploading
f2a4c36 
Signed-off-by: Justin Yu <[email protected]>

Fix lint for storage.py

Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Add a simplified checkpoint manager
49ee126 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Fixes to checkpoint manager
ffa0dd4 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Add unit test for simplified checkpoint manager
15553f7 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Full test coverage
00cc9d7 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Add a simplified checkpoint manager
cb5990e 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Fixes to checkpoint manager
2db9aae 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Add unit test for simplified checkpoint manager
a2067b7 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Support + test resume_from_checkpoint
c7de72a 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Fix result grid bug when no checkpoints saved
db016da 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into air/…
bcbcec9 
…persistence/restore_new_checkpoint_autoft

Signed-off-by: Justin Yu <[email protected]>
@justinvyu
fix merge conflict remainder
d7497e1 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Recover trainable metadata from last_result rather than .tune_metadata
aeb89ba 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Fix restore info log
9556371 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Keep current checkpoint index synchronized on the driver
e897eaa 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Remove checkpoint dirname parsing
3eef417 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into air/…
0e52384 
…persistence/restore_new_checkpoint_autoft
@justinvyu
Update todo comment
89631ab 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Fix lint
d4e20f2 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Merge branch 'air/persistence/restore_new_checkpoint_autoft' into air…
877c4ae 
…/persistence/restore_new_checkpoint_rfc
@justinvyu
Merge branch 'master' of https://github.com/ray-project/ray into air/…
d367e8d 
…persistence/restore_new_checkpoint_rfc

Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Fix e2e test for storage_path=None case
d329e1b 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu
Remove unused code
e382e29 
Signed-off-by: Justin Yu <[email protected]>
@justinvyu justinvyu requested a review from ericl August 8, 2023 20:13
@justinvyu justinvyu marked this pull request as ready for review August 8, 2023 20:13
@justinvyu justinvyu added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Aug 8, 2023
@justinvyu
Copy link
Contributor Author

@ericl Good to merge. #38128 ready for review

@ericl ericl merged commit 55acb74 into ray-project:master Aug 9, 2023
2 checks passed
yutsai84 pushed a commit to yutsai84/ray that referenced this pull request Aug 9, 2023
@justinvyu @yutsai84
[air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…
42ca6fe 
… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Signed-off-by: Yu-Cheng Tsai <[email protected]>
@justinvyu justinvyu deleted the air/persistence/restore_new_checkpoint_rfc branch August 9, 2023 01:15
@ericl ericl mentioned this pull request Aug 10, 2023
Closed
NripeshN pushed a commit to NripeshN/ray that referenced this pull request Aug 15, 2023
@justinvyu @NripeshN
[air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…
f795d0b 
… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Signed-off-by: NripeshN <[email protected]>
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
@justinvyu @arvind-chandra
[air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…
39925a9 
… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Signed-off-by: e428265 <[email protected]>
vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023
@justinvyu
[air] pyarrow.fs persistence (8/n): ray.train.Checkpoint restore:…
a82afa9 
… `resume_from_checkpoint` (ray-project#38143)

This PR supports `Trainer(resume_from_checkpoint)` with the new Checkpoint and adds it as a section of the e2e test.

This PR also fixes a bug where no checkpoints being reported causes the Result object to error on construction.

Signed-off-by: Victor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericl ericl ericl approved these changes

Assignees

@ericl ericl

Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@justinvyu @ericl