[tune] Implement BOHB #5382

lisadunlap · 2019-08-06T03:49:06Z

What do these changes do?

Added TuneBOHB (search), HyperBandForBOHB (early stopping), and an example.

Related issue number

Closes #5100

Linter

I've run scripts/format.sh to lint the changes in this PR.

richardliaw · 2019-08-06T04:03:54Z

python/ray/tune/examples/bohb_example.py

+
+import ray
+from ray.tune import Trainable, run, sample_from
+from ray.tune.schedulers import TuneBOHB, HyperBandForBOHB


TuneBOHB should go to ray.tune.suggest

richardliaw · 2019-08-06T04:04:24Z

python/ray/tune/examples/bohb_example.py

+            config_space,
+            max_concurrent=4,
+            # num_concurrent=100,
+            reward_attr="episode_reward_mean",


reward_attr => metric

richardliaw · 2019-08-06T04:04:52Z

python/ray/tune/schedulers/bohb.py

@@ -0,0 +1,62 @@
+import ray


can you move this file to tune/suggest?

and add the three __future__ imports to the top?

richardliaw · 2019-08-06T04:05:15Z

python/ray/tune/schedulers/bohb.py

+        self.paused = set()
+        self.reward_attr = reward_attr
+        bohb_config = bohb_config or {}
+        self.bohber = BOHB(space, **bohb_config)


oh, we need to import BOHB from HpBandSter

AmplabJenkins · 2019-08-06T13:12:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16023/
Test PASSed.

meyerzinn · 2019-08-08T16:04:35Z

How much more needs to be done before this can be merged? Seems almost complete and it's a major feature to be missing. I'm willing to help if needed. @lisadunlap

richardliaw · 2019-08-09T08:04:17Z

doc/source/tune-searchalg.rst

+
+.. code-block:: python
+
+    tune.run(... , search_alg=TuneBOHB(config_space, ... ))


you also need HyperBand right?

richardliaw · 2019-08-09T08:06:31Z

python/ray/tune/suggest/bohb.py

+from ray.tune.suggest import SuggestionAlgorithm
+
+
+class BOHB(base_config_generator):


ah sorry, I meant “importing BOHB from hpbandster” is ideal

richardliaw · 2019-08-09T08:07:02Z

doc/source/tune-searchalg.rst

@@ -181,6 +182,30 @@ An example of this can be found in `ax_example.py <https://github.com/ray-projec
    :show-inheritance:
    :noindex:

+BOHB (Bayesian optimization and Hyperband)
+---------------------------------------------------


should align with the above string

richardliaw · 2019-08-09T08:16:11Z

python/ray/tune/suggest/bohb.py

+from __future__ import division
+from __future__ import print_function
+
+from hpbandster import BOHB


can you do a soft import here? like in suggest/ax.py; and correspondingly raise an error in init if not found?

richardliaw · 2019-08-09T08:16:23Z

python/ray/tune/suggest/bohb.py

+            self.running.add(trial_id)
+        if "budget" in result.get("hyperband_info", {}):
+            hbs_wrapper = self.to_wrapper(trial_id, result)
+            print("adding new result", vars(hbs_wrapper))


can you remove this?

richardliaw · 2019-08-09T08:16:28Z

python/ray/tune/suggest/bohb.py

+        elif trial_id in self.running:
+            self.running.remove(trial_id)
+        else:
+            import ipdb; ipdb.set_trace()


can you remove this?

python/ray/tune/suggest/bohb.py

richardliaw · 2019-08-09T08:18:24Z

python/ray/tune/schedulers/hb_bohb.py

+class HyperBandForBOHB(HyperBandScheduler):
+    """Implements the HyperBand early stopping algorithm.
+
+    @!!!!!!!!!!!!!! Key changes:


Suggested change

@!!!!!!!!!!!!!! Key changes:

Key changes:

richardliaw · 2019-08-09T08:18:45Z

python/ray/tune/schedulers/hb_bohb.py

+
+
+class HyperBandForBOHB(HyperBandScheduler):
+    """Implements the HyperBand early stopping algorithm.


Suggested change

"""Implements the HyperBand early stopping algorithm.

"""Extends HyperBand early stopping algorithm for BOHB.

richardliaw · 2019-08-09T08:19:18Z

python/ray/tune/schedulers/hb_bohb.py

+            print(Counter([status for _, status in statuses]))
+            trial_runner._search_alg.on_pause(trial.trial_id)
+            return TrialScheduler.PAUSE
+        # import ipdb; ipdb.set_trace()


richardliaw · 2019-08-09T08:19:21Z

python/ray/tune/schedulers/hb_bohb.py

+        if not bracket.filled() or any(
+                status != Trial.PAUSED for t, status in statuses
+                if t is not trial):
+            print(Counter([status for _, status in statuses]))


richardliaw · 2019-08-09T08:22:18Z

python/ray/tune/schedulers/hb_bohb.py

+                bracket.cleanup_full(trial_runner)
+                return TrialScheduler.STOP
+
+            good, bad = bracket.successive_halving(self._reward_attr)


there's no more _reward_attr, so this should be _metric. HyperBand also takes in a mode from its superclass, so I think we just need to do the negation here or something.

richardliaw

Nice! Two last things (in addition to above comments):

Can you add bohb_example as a line in ray/ci/jenkins_tests/run_tune_tests.sh with --smoke-test?

Can you run scripts/format.sh? You'll need flake8 and yapf==0.23.0 and flake8-quotes. This will run formatting.

AmplabJenkins · 2019-08-12T01:35:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16219/
Test FAILed.

AmplabJenkins · 2019-08-12T02:31:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16222/
Test FAILed.

AmplabJenkins · 2019-08-12T04:29:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16225/
Test FAILed.

AmplabJenkins · 2019-08-12T10:04:09Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16229/
Test FAILed.

AmplabJenkins · 2019-08-12T10:27:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16231/
Test FAILed.

AmplabJenkins · 2019-08-12T10:30:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16230/
Test FAILed.

AmplabJenkins · 2019-08-12T10:43:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16232/
Test FAILed.

AmplabJenkins · 2019-08-12T10:50:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16233/
Test FAILed.

AmplabJenkins · 2019-08-12T19:29:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16235/
Test FAILed.

AmplabJenkins · 2019-08-12T20:52:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16238/
Test FAILed.

AmplabJenkins · 2019-08-13T01:28:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16247/
Test PASSed.

AmplabJenkins · 2019-08-13T09:32:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16254/
Test PASSed.

richardliaw · 2019-08-13T20:34:24Z

@20zinnm thanks for offering! This is now merged. Could you try it out and let us know how it goes?

lisadunlap added 3 commits August 1, 2019 19:52

tabular benchmarks

9571fab

added bohb and examples

b3333a1

added bohb and example

0b418b5

richardliaw changed the title ~~Bohb~~ [tune] Implement BOHB Aug 6, 2019

richardliaw self-assigned this Aug 6, 2019

richardliaw reviewed Aug 6, 2019

View reviewed changes

lisadunlap added 2 commits August 8, 2019 23:46

fixed PR changes

5ee0ee1

added reformatted bohb and docs

d6d715b

richardliaw reviewed Aug 9, 2019

View reviewed changes

Delete __init__.py

d69f880

richardliaw reviewed Aug 9, 2019

View reviewed changes

lisadunlap added 3 commits August 9, 2019 01:08

removed BOHB class

00edd95

Merge branch 'bohb' of github.com:lisadunlap/ray into bohb

9679b82

fixed docs

1eb9674

richardliaw reviewed Aug 9, 2019

View reviewed changes

python/ray/tune/suggest/bohb.py Show resolved Hide resolved

richardliaw reviewed Aug 9, 2019

View reviewed changes

richardliaw added 5 commits August 11, 2019 16:18

increase redundancy between HB and BOHB

cc05726

Merge branch 'master' into bohb

9beb26f

Merge branch 'bohb' of github.com:lisadunlap/ray into bohb

b0d81ac

formatting and better example

e34973b

docs

09daf85

richardliaw added 6 commits August 11, 2019 23:43

better tests

b080549

suggest

27f941a

dogstrings

3d42a8b

ok reverse

ce0c1c7

test

bb47d7d

fixexample

1c6e598

richardliaw added 2 commits August 12, 2019 11:10

dockerfile?

77b1d61

fix

44e8dc6

fix

5fec52a

richardliaw approved these changes Aug 12, 2019

View reviewed changes

bohb

bbdaa01

richardliaw merged commit b7d0733 into ray-project:master Aug 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tune] Implement BOHB #5382

[tune] Implement BOHB #5382

lisadunlap commented Aug 6, 2019

richardliaw Aug 6, 2019

richardliaw Aug 6, 2019

richardliaw Aug 6, 2019

richardliaw Aug 6, 2019

AmplabJenkins commented Aug 6, 2019

meyerzinn commented Aug 8, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw Aug 9, 2019

richardliaw left a comment •

edited

Loading

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 13, 2019

AmplabJenkins commented Aug 13, 2019

richardliaw commented Aug 13, 2019


		.. code-block:: python

		tune.run(... , search_alg=TuneBOHB(config_space, ... ))

		from ray.tune.suggest import SuggestionAlgorithm


		class BOHB(base_config_generator):



		class HyperBandForBOHB(HyperBandScheduler):
		"""Implements the HyperBand early stopping algorithm.

	"""Implements the HyperBand early stopping algorithm.
	"""Extends HyperBand early stopping algorithm for BOHB.

[tune] Implement BOHB #5382

[tune] Implement BOHB #5382

Conversation

lisadunlap commented Aug 6, 2019

What do these changes do?

Related issue number

Linter

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Aug 6, 2019

meyerzinn commented Aug 8, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richardliaw left a comment • edited Loading

Choose a reason for hiding this comment

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 13, 2019

AmplabJenkins commented Aug 13, 2019

richardliaw commented Aug 13, 2019

richardliaw left a comment •

edited

Loading