[Feature] [Autoscaler] Scaling Intelligently Based on Observed Resource Bottlenecks (related: task & actor profiling)

[jon-chuang]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

For more efficient resource utilization, I believe the autoscaler, were it to be given a choice in the config over a selection of nodes it could add or remove, each with its own basket of resources, ought to be able to scale intelligently to better address resource bottlenecks based on the real-time observed resource utilization of the recently and presently running tasks, independently or in conjunction with the logical resource requirements of placement groups.

For instance, we might not want to add compute-optimized nodes if the observed bottleneck is memory, and may want to add memory-optimized nodes instead.

Use case

Cost savings for Ray users, and fewer encounters with resource strangling - another great selling point for Ray

Further Work

Related to better resource packing, consider also autoscaler awareness (both logical and profiled) of fractional GPU usage such as MIG (#12413, NVIDIA MIG blogpost)

Related issues

Memory aware scheduling & profiling-based placement: #20495

maybe related (autoscaler's poor understanding of resources): #20476

Related to task and actor profiling which will be written up an RFC soon

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[sivaguru-pat]

@ericl @jon-chuang is it on the road map?

[pang-wu]

+1, any update on this?

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[stale]

Hi again! The issue will be closed because there has been no more activity in the 14 days since the last message.

Please feel free to reopen or open a new issue if you'd still like it to be addressed.

Again, you can always ask for help on our discussion forum or Ray's public slack channel.

Thanks again for opening the issue!