Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Tune] Add timeout for experiment checkpoint syncing to cloud #30855

Merged
merged 17 commits into from
Dec 7, 2022
Merged

[Tune] Add timeout for experiment checkpoint syncing to cloud #30855

merged 17 commits into from
Dec 7, 2022

Commits on Nov 30, 2022

  1. Add sync timeout to Syncer 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    b8e4f40 View commit details
    Browse the repository at this point in the history

  2. Add unit test skeleton 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    7d56179 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2022

  1. Add logic for setting timeout based on sync process start time 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 1, 2022
    Configuration menu
    Copy the full SHA
    a30021c View commit details
    Browse the repository at this point in the history

  2. Fix existing sync handling to use Syncer.wait, which sets a timeout 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 1, 2022
    Configuration menu
    Copy the full SHA
    6dd30ba View commit details
    Browse the repository at this point in the history

  3. Move time remaining logic to the BackgroundProcess wait 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 1, 2022
    Configuration menu
    Copy the full SHA
    b73ec5a View commit details
    Browse the repository at this point in the history

  4. Test that new syncs should start after timeout 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 1, 2022
    Configuration menu
    Copy the full SHA
    d3f2c0f View commit details
    Browse the repository at this point in the history

  5. Set sync_timeout config in _DefaultSyncer 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 1, 2022
    Configuration menu
    Copy the full SHA
    c9e02e3 View commit details
    Browse the repository at this point in the history

  6. Add sync timeout handling to TrialRunner 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 1, 2022
    Configuration menu
    Copy the full SHA
    b81beca View commit details
    Browse the repository at this point in the history

  7. Add trial runner checkpoint sync timeout tests 

    Signed-off-by: Justin Yu <[email protected]>

Remove unused pytest import

Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 1, 2022
    Configuration menu
    Copy the full SHA
    5c79af4 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2022

  1. Merge branch 'master' of https://github.com/ray-project/ray into expe… 

    …riment_checkpoint_timeout
    @justinvyu
    justinvyu committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    23bdbb2 View commit details
    Browse the repository at this point in the history

  2. Update expected exception type in test 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    a0fc9fc View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of https://github.com/ray-project/ray into expe… 

    …riment_checkpoint_timeout
    @justinvyu
    justinvyu committed Dec 2, 2022
    Configuration menu
    Copy the full SHA
    064b2a7 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2022

  1. Add start_time property to mock sync process in test 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    dc26d90 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/ray-project/ray into expe… 

    …riment_checkpoint_timeout
    @justinvyu
    justinvyu committed Dec 5, 2022
    Configuration menu
    Copy the full SHA
    c588bf8 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2022

  1. Add comments 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    576acd4 View commit details
    Browse the repository at this point in the history

  2. Add docstring for syncer args 

    Signed-off-by: Justin Yu <[email protected]>
    @justinvyu
    justinvyu committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    d6b6ca2 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of https://github.com/ray-project/ray into expe… 

    …riment_checkpoint_timeout
    @justinvyu
    justinvyu committed Dec 6, 2022
    Configuration menu
    Copy the full SHA
    e531744 View commit details
    Browse the repository at this point in the history