[release][nightly][pg] tune_cifar_pytorch_pbt_example does not work on nightly (Placement group error)

[AmeerHajAli]

Search before asking

• I searched the issues and found no similar issues.

Ray Component

Ray Tune

What happened + What you expected to happen

I ran with ray 1.8 it works, when I run with commit c0aeb4a I get the following error:

(run pid=864) ray::TrainTrainable.train_buffered() (pid=253, ip=10.0.24.128, repr=<ray.train.trainer.TrainTrainable object at 0x7efea85879d0>)
(run pid=864)   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 262, in run
(run pid=864)     self._entrypoint()
(run pid=864)   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/function_runner.py", line 331, in entrypoint
(run pid=864)     self._status_reporter.get_checkpoint())
(run pid=864)   File "/Users/ameerhajali/anaconda3/envs/ray/lib/python3.7/site-packages/ray/tune/function_runner.py", line 597, in _trainable_func
(run pid=864)   File "/Users/ameerhajali/anaconda3/envs/ray/lib/python3.7/site-packages/ray/train/trainer.py", line 761, in tune_function
(run pid=864)   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/train/trainer.py", line 162, in __init__
(run pid=864)     max_retries=max_retries)
(run pid=864)   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/actor.py", line 524, in remote
(run pid=864)     runtime_env=new_runtime_env)
(run pid=864)   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/actor.py", line 747, in _remote
(run pid=864)     placement_group=placement_group)
(run pid=864)   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/placement_group.py", line 452, in configure_placement_group_based_on_context
(run pid=864)     placement_resources, task_or_actor_repr)
(run pid=864)   File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/util/placement_group.py", line 370, in _validate_resource_shape
(run pid=864)     raise ValueError(f"Cannot schedule {task_or_actor_repr} with "
(run pid=864) ValueError: Cannot schedule BackendExecutor with the placement group because the resource request {'node:10.0.24.128': 0.01, 'CPU': 0} cannot fit into any bundles for the placement group, [{'CPU': 1.0}, {'node:10.0.16.137': 0.01}, {'CPU': 1.0}, {'CPU': 1.0}].

Versions / Dependencies

python 3.7, ray c0aeb4a (nightly)

Reproduction script

RAY_ADDRESS=anyscale://ga-demo-aws python tune_cifar_pytorch_pbt_example.py

The cluster env:

Anything else

@richardliaw , can you please help with this one?

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[AmeerHajAli]

@scv119 / @ericl do you know what can be going wrong?

[AmeerHajAli]

Looks like this commit broke it: #20123
Before that it works.

[rkooo567]

The error message shows what’s happening here? It tries to schedule an actor to the bundle that doesn’t have requested resources. (The node resources .128 is not in any bundle). Should be the application error (that it configures resources incorrectly)