Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tune] Change the log syncing behavior #4450

Merged
merged 48 commits into from
Jul 3, 2019
Merged

[tune] Change the log syncing behavior #4450

merged 48 commits into from
Jul 3, 2019

Conversation

hartikainen
Copy link
Contributor

@hartikainen hartikainen commented Mar 21, 2019 •
edited by richardliaw
Loading

What do these changes do?

Refactor the log sync behavior.

TODOs:

  • Fix up Trial syncing (remove remote capabilities from Trial?)

    • Allow custom sync_function for remote to driver syncing. This may require a bit of restructuring to LogSyncing as a mixin.

  • Write Tests:

    • Node - Node sync_string (use something like "echo")
    • Node - Node sync_function
    • Local - Cloud sync_string (use something like "echo")
    • Local - Cloud sync_function
    • [~] Test Migration of files (Change the local ip of the head node in test_cluster.py). - Punting on this one because it is captured in e2e ft test.
    • Write instructions for an end-to-end fault tolerance test - I have one but not sure where to put it.

  • os.path.expanduser on remote node (this is done in [tune] Later expansion of local_dir #4806)

  • Docs

  • DeprecationWarning

  • Tests

Related issue number

@richardliaw

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13169/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13545/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13575/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/13576/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14020/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14021/
Test FAILed.

@richardliaw richardliaw self-assigned this May 2, 2019
@richardliaw richardliaw mentioned this pull request May 10, 2019
Closed
@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14254/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14260/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14262/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14264/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14274/
Test FAILed.

@richardliaw richardliaw added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Jul 1, 2019
@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15004/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15003/
Test PASSed.

hartikainen
hartikainen commented Jul 2, 2019
View reviewed changes
doc/source/tune-usage.rst Outdated
@@ -259,7 +259,7 @@ of a trial, you can additionally set the checkpoint_at_end to True. An example i
Recovering From Failures (Experimental)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Tune automatically persists the progress of your experiments, so if an experiment crashes or is otherwise cancelled, it can be resumed with ``resume=True``. The default setting of ``resume=False`` creates a new experiment, and ``resume="prompt"`` will cause Tune to prompt you for whether you want to resume. You can always force a new experiment to be created by changing the experiment name.
Tune automatically persists the progress of your experiments, so if an experiment crashes or is otherwise cancelled, it can be resumed by passing one of True, False, "LOCAL", "REMOTE", or "PROMPT" to ``tune.run(resume=...)``. The default setting of ``resume=False`` creates a new experiment. ``resume="LOCAL"`` and ``resume=True`` restore the experiment from ``local_dir/[experiment_name]``. ``resume="REMOTE"`` syncs the upload dir down to the local dir and then restore the experiment from ``local_dir/experiment_name``. ``resume="PROMPT"`` will cause Tune to prompt you for whether you want to resume. You can always force a new experiment to be created by changing the experiment name.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Tune automatically persists the progress of your experiments, so if an experiment crashes or is otherwise cancelled, it can be resumed by passing one of True, False, "LOCAL", "REMOTE", or "PROMPT" to ``tune.run(resume=...)``. The default setting of ``resume=False`` creates a new experiment. ``resume="LOCAL"`` and ``resume=True`` restore the experiment from ``local_dir/[experiment_name]``. ``resume="REMOTE"`` syncs the upload dir down to the local dir and then restore the experiment from ``local_dir/experiment_name``. ``resume="PROMPT"`` will cause Tune to prompt you for whether you want to resume. You can always force a new experiment to be created by changing the experiment name.
Tune automatically persists the progress of your experiments, so if an experiment crashes or is otherwise cancelled, it can be resumed by passing one of True, False, "LOCAL", "REMOTE", or "PROMPT" to ``tune.run(resume=...)``. The default setting of ``resume=False`` creates a new experiment. ``resume="LOCAL"`` and ``resume=True`` restore the experiment from ``local_dir/[experiment_name]``. ``resume="REMOTE"`` syncs the upload dir down to the local dir and then restores the experiment from ``local_dir/experiment_name``. ``resume="PROMPT"`` will cause Tune to prompt you for whether you want to resume. You can always force a new experiment to be created by changing the experiment name.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch

@hartikainen
Copy link
Contributor Author

Looks good to me!

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15023/
Test PASSed.

@richardliaw
Copy link
Contributor

Awesome!

richardliaw
richardliaw reviewed Jul 2, 2019
View reviewed changes
python/ray/tune/tune.py Outdated
else:
logger.info("Tip: to resume incomplete experiments, "
"pass resume='prompt' or resume=True to run()")
def _get_resume_path(local_checkpoint_dir, remote_checkpoint_dir):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is extraneous

richardliaw
richardliaw reviewed Jul 2, 2019
View reviewed changes
python/ray/tune/trial_runner.py Outdated
def _validate_resume(self, resume_type):
"""
Args:
resume_type: One of "REMOTE", "LOCAL", "PROMPT".
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
resume_type: One of "REMOTE", "LOCAL", "PROMPT".
resume_type: One of "REMOTE", "LOCAL", True, "PROMPT".

richardliaw
richardliaw reviewed Jul 2, 2019
View reviewed changes
python/ray/tune/trial_runner.py Outdated
self._metadata_checkpoint_dir = metadata_checkpoint_dir
self._local_checkpoint_dir = local_checkpoint_dir

# TODO(rliaw): This may fail
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# TODO(rliaw): This may fail

richardliaw
richardliaw reviewed Jul 2, 2019
View reviewed changes
python/ray/tune/syncer.py Outdated
Args:
local_dir: Source directory for syncing.
remote_dir: Target directory for syncing. If None,
returns NoopSyncer.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
returns NoopSyncer.
returns BaseSyncer with a noop.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15030/
Test PASSed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15032/
Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15040/
Test FAILed.

@richardliaw richardliaw merged commit 9e0192b into ray-project:master Jul 3, 2019
@hartikainen hartikainen deleted the bunch-of-log-sync-fixes branch July 3, 2019 04:36
This was referenced Jul 9, 2019
Closed
Closed
Closed
Closed
lscheinkman added a commit to lscheinkman/nupic.research that referenced this pull request Jan 8, 2021
@lscheinkman
Fix ray.tune api change. See ray-project/ray#4450
3699497
@lscheinkman lscheinkman mentioned this pull request Jan 8, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@richardliaw richardliaw richardliaw approved these changes

Assignees

@richardliaw richardliaw

Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hartikainen @AmplabJenkins @richardliaw