Added example to user guide for cloud checkpointing #20045

worldveil · 2021-11-04T01:18:24Z

Added a few starter code examples just to get the ball rolling in documenting cloud sync and trainables.

worldveil · 2021-11-04T19:09:29Z

doc/source/tune/user-guide.rst

+        # when checkpoints happen, all the trained parameters are saved alongside, 
+        # not just your model's hyperparameters! useful if you want your checkpoints 
+        # to be "inference ready" checkpoints of your models
+        tune.with_parameters(my_pytorch_training_func),


Suggested change

tune.with_parameters(my_pytorch_training_func),

tune.with_parameters(tune.durable(my_pytorch_training_func)),

doc/source/tune/user-guide.rst

worldveil · 2021-11-04T19:12:22Z

doc/source/tune/user-guide.rst

+    # cluster when your instances are terminated
+    sync_config = tune.syncConfig(
+        upload_dir="s3://my-checkpoints-bucket/path/",  # requires AWS credentials
+        sync_to_cloud=True  # this can also be a func for custom logic!


sync_to_driver=False

worldveil · 2021-11-04T19:14:40Z

doc/source/tune/user-guide.rst


-    tune.run(train_func)
+* Locally: ``local_dir/my-tune-exp/<trial_name>/checkpoint_<step>``


if sync_to_driver=False AND running locally from laptop → cluster, then this NOT stored locally

if sync_to_driver=False AND run with ray submit, then only trials run on head will be on head

doc/source/tune/user-guide.rst

worldveil · 2021-11-04T19:28:21Z

doc/source/tune/user-guide.rst

+
+        # our same config as above!
+        restore=sync_config,
+    )

 .. _tune-distributed-checkpointing:



Could use this example:
https://github.com/ray-project/ray/blob/master/python/ray/tune/examples/xgboost_example.py

worldveil · 2021-11-06T00:32:11Z

doc/source/tune/user-guide.rst

+
+   2. **Custom training function**
+
+      * All this means is that your function has to expose a ``checkpoint_dir`` argument in the function signature, and call ``tune.checkpoint_dir``. See this example, it's quite simple to do.


I want to link here to custom_func_checkpointing.rst, but not sure how to do that

worldveil · 2021-11-06T00:32:39Z

doc/source/tune/user-guide.rst

+A simple local/rsync checkpointing example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Local or rsync checkpointing can be a good option if:


Double check me on this list!

This looks fine. I guess you can also add "if you are not interested in persistence"

worldveil · 2021-11-06T00:33:03Z

doc/source/tune/user-guide.rst

@@ -346,7 +454,7 @@ Distributed Checkpointing

 On a multinode cluster, Tune automatically creates a copy of all trial checkpoints on the head node. This requires the Ray cluster to be started with the :ref:`cluster launcher <cluster-cloud>` and also requires rsync to be installed.

-Note that you must use the ``tune.checkpoint_dir`` API to trigger syncing.
+Note that you must use the ``tune.checkpoint_dir`` API to trigger syncing (or use a Ray built-in model type described here).


Also want to link to that example custom_func_checkpointing.rst or a better one from github.

worldveil · 2021-11-06T00:34:40Z

@Yard1 made a bunch of fixes & adds here.

Would love your review, esp among some of the examples I linked. We might have better ones to link to for the "ray models" bullet list

Yard1

I have pushed a commit that improved the way links are handled, and added links to custom_func_checkpointing as you wanted - let me know if anything is unclear

doc/source/tune/user-guide.rst

Co-authored-by: Antoni Baum <[email protected]>

krfricke · 2021-11-08T10:29:33Z

doc/source/tune/user-guide.rst

+A simple (cloud) checkpointing example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Cloud storage-backed Tune checkpointing is the recommended best practice for both performance and reliability reasons. rsync-based checkpointing is also incompatible with Ray on Kubernetes. If you'd rather checkpoint locally or use rsync based checkpointing, see here.


Link missing ("here")

also, please add a sentence of context here (explain ssh based rsync checkpointing) - it may not be clear for users that this is the default way.

I'd also rather frame this positively ("it also enables checkpointing if using Ray on Kubernetes, which does not work out of the box" or so).

How do I link to an anchor? "here" would ideally link to the A simple local/rsync checkpointing example section below

Adjusted the text, let me know if phrased better @krfricke

doc/source/tune/user-guide.rst

Co-authored-by: Antoni Baum <[email protected]>

…tune-checkpointing

worldveil · 2021-11-09T06:27:51Z

@krfricke @Yard1 hopefully should have addressed all your comments

worldveil · 2021-11-09T06:16:59Z

doc/source/tune/user-guide.rst

+A simple (cloud) checkpointing example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Cloud storage-backed Tune checkpointing is the recommended best practice for both performance and reliability reasons. rsync-based checkpointing is also incompatible with Ray on Kubernetes. If you'd rather checkpoint locally or use rsync based checkpointing, see here.


Adjusted the text, let me know if phrased better @krfricke

doc/source/tune/user-guide.rst

worldveil · 2021-11-10T19:05:28Z

doc/source/tune/user-guide.rst

-In this example, checkpoints will be saved by training iteration to ``local_dir/exp_name/trial_name/checkpoint_<step>``.
+* **Locally**: not saved! we set ``sync_to_driver=False``, so nothing will be sync'd to the driver (your laptop)
+* **S3**: ``s3://my-checkpoints-bucket/path/my-tune-exp/<trial_name>/checkpoint_<step>``
+* **On head node**: ``~/ray-results/my-tune-exp/<trial_name>/checkpoint_<step>``


Suggested change

* **On head node**: ``~/ray-results/my-tune-exp/<trial_name>/checkpoint_<step>``

* **On head node**: ``~/ray-results/my-tune-exp/<trial_name>/checkpoint_<step>`` (will only have trials run on the head node)

@Yard1 added suggestion

worldveil · 2021-11-10T19:09:53Z

@krfricke all look good here? ready to merge unless you have additional comments

krfricke

two small changes

krfricke · 2021-11-11T17:17:35Z

doc/source/tune/user-guide.rst

+A simple (cloud) checkpointing example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Cloud storage-backed Tune checkpointing is the recommended best practice for both performance and reliability reasons. It also enables checkpointing if using Ray on Kubernetes, which does not work out of the box with rsync-based sync, which relies on SSH. If you'd rather checkpoint locally or use rsync based checkpointing, see here.


"here" - link missing?

krfricke · 2021-11-11T17:20:03Z

doc/source/tune/user-guide.rst

+    # with_parameters() ensures when we checkpoint, weights are saved (not just hyperparameters)
+    durable_trainer_with_params = tune.with_parameters(my_trainable)


Ah this is not correct - let's not use with_parameters here at all

Generally, with_parameters is used to e.g. re-use data across trainables. It saves parameters in the object store to be re-used, similarly to functools.partial - we don't use this ere, so let's just directly pass the trainable below

…nting

doc/source/tune/examples/custom_func_checkpointing.rst

krfricke

I adjusted the example to match the latest API changes and moved the docs around a bit

Added example to user guide for cloud checkpointing

fecc4d0

Yard1 self-assigned this Nov 4, 2021

Yard1 self-requested a review November 4, 2021 16:16

worldveil commented Nov 4, 2021

View reviewed changes

doc/source/tune/user-guide.rst Show resolved Hide resolved

worldveil commented Nov 4, 2021

View reviewed changes

doc/source/tune/user-guide.rst Show resolved Hide resolved

worldveil commented Nov 4, 2021

View reviewed changes

will added 3 commits November 5, 2021 16:02

Added Tune example

7729b3d

Added Tune example, extended

5725329

Couple more nits

8d86780

worldveil commented Nov 6, 2021

View reviewed changes

worldveil mentioned this pull request Nov 6, 2021

Update RLlib in 60 seconds example #20122

Closed

Improve example links

911a77f

Yard1 requested changes Nov 8, 2021

View reviewed changes

doc/source/tune/user-guide.rst Outdated Show resolved Hide resolved

doc/source/tune/user-guide.rst Outdated Show resolved Hide resolved

doc/source/tune/user-guide.rst Outdated Show resolved Hide resolved

doc/source/tune/user-guide.rst Outdated Show resolved Hide resolved

Apply suggestions from code review

58a0c77

Co-authored-by: Antoni Baum <[email protected]>

krfricke reviewed Nov 8, 2021

View reviewed changes

worldveil and others added 5 commits November 8, 2021 22:18

Update doc/source/tune/user-guide.rst

abb9f97

Co-authored-by: Antoni Baum <[email protected]>

Made adjustments for comments

298238a

Merge branch 'tune-checkpointing' of github.com:ray-project/ray into …

61aca90

…tune-checkpointing

More adjustments from comments

71acfd2

Changed two examples to be the same comments

7dd9f5b

ray-project deleted a comment from Yard1 Nov 10, 2021

worldveil commented Nov 10, 2021

View reviewed changes

Yard1 approved these changes Nov 10, 2021

View reviewed changes

Update doc/source/tune/user-guide.rst

1c739cd

krfricke self-assigned this Nov 11, 2021

krfricke requested changes Nov 11, 2021

View reviewed changes

worldveil changed the title ~~[WIP] Added example to user guide for cloud checkpointing~~ Added example to user guide for cloud checkpointing Nov 12, 2021

Address comments

2b289e5

Yard1 approved these changes Nov 12, 2021

View reviewed changes

Kai Fricke added 2 commits November 12, 2021 09:40

Merge remote-tracking branch 'upstream/master' into doc/tune-checkpoi…

fe05492

…nting

Integrate with previous doc changes

c681e0a

krfricke reviewed Nov 12, 2021

View reviewed changes

doc/source/tune/examples/custom_func_checkpointing.rst Outdated Show resolved Hide resolved

krfricke reviewed Nov 12, 2021

View reviewed changes

Yard1 approved these changes Nov 12, 2021

View reviewed changes

Yard1 added 4 commits November 12, 2021 11:27

Add example

2b52ee2

Merge branch 'master' into tune-checkpointing

26bb323

Lint

73ae3d6

Merge branch 'master' into tune-checkpointing

0d51da6

krfricke approved these changes Nov 15, 2021

View reviewed changes

krfricke merged commit fa878e2 into master Nov 15, 2021

krfricke deleted the tune-checkpointing branch November 15, 2021 15:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added example to user guide for cloud checkpointing #20045

Added example to user guide for cloud checkpointing #20045

worldveil commented Nov 4, 2021 •

edited

Loading

worldveil Nov 4, 2021 •

edited

Loading

worldveil Nov 4, 2021

worldveil Nov 4, 2021

worldveil Nov 4, 2021

worldveil Nov 6, 2021

worldveil Nov 6, 2021

Yard1 Nov 8, 2021

worldveil Nov 6, 2021

worldveil commented Nov 6, 2021

Yard1 left a comment

krfricke Nov 8, 2021

worldveil Nov 9, 2021

worldveil Nov 9, 2021

worldveil commented Nov 9, 2021

worldveil Nov 9, 2021

worldveil Nov 10, 2021

worldveil Nov 10, 2021

worldveil commented Nov 10, 2021

krfricke left a comment

krfricke Nov 11, 2021

krfricke Nov 11, 2021

krfricke left a comment

	tune.with_parameters(my_pytorch_training_func),
	tune.with_parameters(tune.durable(my_pytorch_training_func)),


		tune.run(train_func)
		* Locally: ``local_dir/my-tune-exp/<trial_name>/checkpoint_<step>``


		2. Custom training function

		* All this means is that your function has to expose a ``checkpoint_dir`` argument in the function signature, and call ``tune.checkpoint_dir``. See this example, it's quite simple to do.

	* On head node: ``~/ray-results/my-tune-exp/<trial_name>/checkpoint_<step>``
	* On head node: ``~/ray-results/my-tune-exp/<trial_name>/checkpoint_<step>`` (will only have trials run on the head node)

		# with_parameters() ensures when we checkpoint, weights are saved (not just hyperparameters)
		durable_trainer_with_params = tune.with_parameters(my_trainable)

Added example to user guide for cloud checkpointing #20045

Added example to user guide for cloud checkpointing #20045

Conversation

worldveil commented Nov 4, 2021 • edited Loading

worldveil Nov 4, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

worldveil commented Nov 6, 2021

Yard1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

worldveil commented Nov 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

worldveil commented Nov 10, 2021

krfricke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krfricke left a comment

Choose a reason for hiding this comment

worldveil commented Nov 4, 2021 •

edited

Loading

worldveil Nov 4, 2021 •

edited

Loading