[tune] Test example checkpointing #4728

richardliaw · 2019-05-01T21:19:35Z

What do these changes do?

Due to the rmtree of save_to_object, TF saving fails. This PR fixes that.

Related issue number

Linter

I've run scripts/format.sh to lint the changes in this PR.

AmplabJenkins · 2019-05-01T21:21:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/648/
Test PASSed.

AmplabJenkins · 2019-05-01T22:31:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/649/
Test PASSed.

AmplabJenkins · 2019-05-01T22:38:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/650/
Test PASSed.

AmplabJenkins · 2019-05-01T23:15:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14040/
Test FAILed.

AmplabJenkins · 2019-05-01T23:17:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14044/
Test FAILed.

AmplabJenkins · 2019-05-02T00:35:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14043/
Test FAILed.

AmplabJenkins · 2019-05-02T05:06:10Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/655/
Test PASSed.

AmplabJenkins · 2019-05-02T05:12:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/656/
Test PASSed.

AmplabJenkins · 2019-05-02T05:50:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14055/
Test FAILed.

richardliaw · 2019-05-02T05:56:08Z

@hartikainen this is also related to your PR

AmplabJenkins · 2019-05-02T06:54:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14056/
Test FAILed.

python/ray/tune/trainable.py

ericl · 2019-05-05T21:52:46Z

python/ray/tune/trainable.py

+        tmpdir = info["savedir"]
+        if not os.path.exists(tmpdir):
+            # We recreate the same subtree and path.
+            os.makedirs(tmpdir)


Why change this logic? This will try to recreate the absolute path right? Whereas, the previous code recreated it relative to the given tempdir.

So this was one way to enable tf.Saver functionality.

saver = tf.Saver() prefix = self.saver.save(self.sess, prefix=checkpoint_dir + "/save", global_step=self.iterations)

tf.Saver saves things into a given directory with this format:

╰─ ls /Users/rliaw/ray_results/mnist_hyperband_test/TrainMNIST_0_learning_rate=0.00022789_2019-05-01_15-21-20lzl3rbbe/tmpudqrk34zsave_to_object/checkpoint_2/ checkpoint save-2.data-00000-of-00001 save-2.index save-2.meta save-2.tune_metadata

And, in checkpoint, it saves absolute paths:

╰─ cat /Users/rliaw/ray_results/mnist_hyperband_test/TrainMNIST_0_learning_rate=0.00022789_2019-05-01_15-21-20lzl3rbbe/tmpudqrk34zsave_to_object/checkpoint_2/checkpoint model_checkpoint_path: "/Users/rliaw/ray_results/mnist_hyperband_test/TrainMNIST_0_learning_rate=0.00022789_2019-05-01_15-21-20lzl3rbbe/tmpudqrk34zsave_to_object/checkpoint_2/save-2" all_model_checkpoint_paths: "/Users/rliaw/ray_results/mnist_hyperband_test/TrainMNIST_0_learning_rate=0.00022789_2019-05-01_15-21-20lzl3rbbe/tmpudqrk34zsave_to_object/checkpoint_2/save-2"

There's alternative workarounds for this problem (i.e., doing a os.chdir in trainable:save_to_object()), but this seemed reasonable...

Won't this break when I move it to some other machine which does have /Users/rliaw?

AmplabJenkins · 2019-05-09T02:18:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/743/
Test PASSed.

AmplabJenkins · 2019-05-09T05:02:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14156/
Test FAILed.

ericl

Looks fine but I would still raise a ValueError on unexpected return

AmplabJenkins · 2019-07-07T10:10:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1487/
Test PASSed.

AmplabJenkins · 2019-07-07T12:56:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1488/
Test FAILed.

AmplabJenkins · 2019-07-07T21:11:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1492/
Test FAILed.

AmplabJenkins · 2019-07-08T07:41:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15180/
Test PASSed.

AmplabJenkins · 2019-07-08T11:51:00Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15192/
Test PASSed.

AmplabJenkins · 2019-07-08T19:16:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1518/
Test FAILed.

AmplabJenkins · 2019-07-08T20:44:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15210/
Test FAILed.

AmplabJenkins · 2019-07-08T20:52:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1519/
Test FAILed.

AmplabJenkins · 2019-07-08T22:11:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1521/
Test FAILed.

AmplabJenkins · 2019-07-08T22:51:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15211/
Test FAILed.

AmplabJenkins · 2019-07-08T23:56:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15213/
Test FAILed.

AmplabJenkins · 2019-07-09T01:09:04Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15216/
Test FAILed.

AmplabJenkins · 2019-07-09T05:07:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1524/
Test FAILed.

AmplabJenkins · 2019-07-10T07:33:04Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1572/
Test FAILed.

AmplabJenkins · 2019-07-10T07:36:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15262/
Test PASSed.

richardliaw added 2 commits April 3, 2019 10:41

partial fix - somehow file is still not found

2bf3b95

workermode

2df21d1

richardliaw added 2 commits May 1, 2019 15:28

Trainable workaround

d14ebfe

Merge branch 'master' into fix_tune_tf_checkpoint

c7a2bcd

Some changes

d173fed

richardliaw added 3 commits May 2, 2019 01:03

remove comments

5418861

Merge branch 'master' into fix_tune_tf_checkpoint

d6e5339

nit

151f821

formatting

2cbf9c4

richardliaw changed the title ~~(wip)[tune] Fix TF checkpoint for pausing~~ [tune] Fix TF checkpoint for pausing May 2, 2019

richardliaw assigned ericl May 2, 2019

ericl reviewed May 5, 2019

View reviewed changes

python/ray/tune/trainable.py Outdated Show resolved Hide resolved

ericl reviewed May 5, 2019

View reviewed changes

python/ray/tune/trainable.py Show resolved Hide resolved

ericl reviewed May 5, 2019

View reviewed changes

richardliaw mentioned this pull request May 8, 2019

[tune] Use os.path.join for .tune_metadata path #4722

Closed

richardliaw added 2 commits May 8, 2019 18:45

stashpop

c5da414

better test'

a55e5df

ericl approved these changes Jul 7, 2019

View reviewed changes

richardliaw added 2 commits July 7, 2019 21:03

Merge branch 'master' into fix_tune_tf_checkpoint

d494cf9

faster

121a178

valueerr

2a7e26b

move test to jenkins

239e67d

fix tune test docker

4679052

fix tests

dba6dad

richardliaw added 2 commits July 8, 2019 17:44

downloads

f51ea71

tmpdir

899a792

richardliaw added 2 commits July 9, 2019 20:00

fix_pytorch

781d6db

load model

9b67b9e

richardliaw merged commit 0b540ab into ray-project:master Jul 10, 2019

richardliaw deleted the fix_tune_tf_checkpoint branch July 10, 2019 08:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tune] Test example checkpointing #4728

[tune] Test example checkpointing #4728

richardliaw commented May 1, 2019 •

edited

Loading

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 2, 2019

AmplabJenkins commented May 2, 2019

AmplabJenkins commented May 2, 2019

AmplabJenkins commented May 2, 2019

richardliaw commented May 2, 2019

AmplabJenkins commented May 2, 2019

ericl May 5, 2019

richardliaw May 9, 2019 •

edited

Loading

ericl May 16, 2019

AmplabJenkins commented May 9, 2019

AmplabJenkins commented May 9, 2019

ericl left a comment

AmplabJenkins commented Jul 7, 2019

AmplabJenkins commented Jul 7, 2019

AmplabJenkins commented Jul 7, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 9, 2019

AmplabJenkins commented Jul 9, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

[tune] Test example checkpointing #4728

[tune] Test example checkpointing #4728

Conversation

richardliaw commented May 1, 2019 • edited Loading

What do these changes do?

Related issue number

Linter

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 2, 2019

AmplabJenkins commented May 2, 2019

AmplabJenkins commented May 2, 2019

AmplabJenkins commented May 2, 2019

richardliaw commented May 2, 2019

AmplabJenkins commented May 2, 2019

ericl May 5, 2019

Choose a reason for hiding this comment

richardliaw May 9, 2019 • edited Loading

Choose a reason for hiding this comment

ericl May 16, 2019

Choose a reason for hiding this comment

AmplabJenkins commented May 9, 2019

AmplabJenkins commented May 9, 2019

ericl left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Jul 7, 2019

AmplabJenkins commented Jul 7, 2019

AmplabJenkins commented Jul 7, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 8, 2019

AmplabJenkins commented Jul 9, 2019

AmplabJenkins commented Jul 9, 2019

AmplabJenkins commented Jul 10, 2019

AmplabJenkins commented Jul 10, 2019

richardliaw commented May 1, 2019 •

edited

Loading

richardliaw May 9, 2019 •

edited

Loading