[Serve] Optimize the _get_live_deployments function #45793
Labels
enhancement
Request for new feature and/or capability
P1
Issue that should be fixed within a few weeks
serve
Ray Serve Related Issue
Description
The _get_live_deployments function internally calls get_deployments_in_application that iterates over
self._deployment_state
( that stores all the live deployments) and filters out deployment for a specific app. This is inefficient especially when you have a large of serve application running. (For every serve application we are iterating over all the deployments)May be we can optimize it by defining a new dictionary (in place of
self._deployment_state
) that maps app name to the live deployments name. However I don't know much about the ray internals and their could be a better way around this ( we can rethink the class structures)Use case
Our system is running a very large number of DeploymentHandles (see #44784 for more details). We've noticed that the Serve controller gets overloaded (>100% CPU usage) and a small percentage of this CPU usage is coming from the
_get_live_deployments
(get_deployments_in_application) function, which is called at multiple places.The text was updated successfully, but these errors were encountered: