Raise an Exception when a task, actor, or placement group is permanently infeasible

[ericl]

After #18724, we should be able to raise exceptions when a task, actor, or placement group becomes permanently infeasible. To do this, the autoscaler can periodically publish a list of "permanently infeasible" resource demands to the GCS via an RPC, and this list can be distributed across the cluster in resource poll requests from the GCS.

The errors raised would be as follows:

@ray.remote(num_gpus=999)
class A:
   def f(self):
      pass

@ray.remote(num_gpus=999)
def f():
   pass

ray.get(f.remote())
# -> raises UnschedulableError("No available node types can fulfill request {"GPU": 999}.")

a = A.remote()
ray.get(a.f.remote())
# -> raises UnschedulableError("No available node types can fulfill request {"GPU": 999}.")

pg = ray.placement_group([{"GPU": 999}])
ray.get(pg.ready())
# -> raises UnschedulableError("The cluster configuration cannot fulfill [{"GPU": 999}].")

ray.get(f.options(placement_group=pg).remote())
# -> raises UnschedulableError("The cluster configuration cannot fulfill [{"GPU": 999}].")

PRD doc: https://docs.google.com/document/d/1OT6m4xQDN8UtsBgnAMpX6nhXpNAfdeHJVve-iGhw1WI/edit

cc @edoakes @scv119 @richardliaw @stephanie-wang @rkooo567

[rkooo567]

pg = ray.placement_group([{"GPU": 999}])
ray.get(pg.ready())
# -> raises UnschedulableError("The cluster configuration cannot fulfill [{"GPU": 999}].")

@ericl do you also plan to mark "placement groups that are already deleted" as infeasible?

[ericl]

Yes we should do that too.

…

On Mon, Sep 27, 2021, 5:49 AM SangBin Cho ***@***.***> wrote: pg = ray.placement_group([{"GPU": 999}]) ray.get(pg.ready()) # -> raises UnschedulableError("The cluster configuration cannot fulfill [{"GPU": 999}].") @ericl <https://github.com/ericl> do you also plan to mark "placement groups that are already deleted" as infeasible? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18835 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADUSRAXHXKVNQPCMSA6JDUEBR6RANCNFSM5ESQDWUQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[rkooo567]

Btw, when do we plan to do this task?

[ericl]

@rkooo567 you're currently working on a design for this right? cc @richardliaw

[rkooo567]

Yes. I've been putting it off a little bit (focus on other tasks first) because of the feature freeze. But I am planning to have a concrete proposal by the end of the sprint and work on it next sprint

[rkooo567]

The initial draft; https://docs.google.com/document/d/158cRgisVt6JZ55ARckrRHDLjLJ6YA_9lnzN8bp9F6vU/edit

[xwjiang2010]

Hi @rkooo567 wondering if there is an update on this ticket? There is some feedback about insufficient resource messages can be confusing if autoscaler is enabled.

[ericl]

I think this is blocked on pending work from @wuisawesome refactoring the autoscaler interfaces.