[Serve] ray_serve_deployment_queued_queries doesn't handle client disconnects

[shrekris-anyscale]

What happened + What you expected to happen

The ray_serve_deployment_queued_queries metric tracks the number of ServeHandle requests that are queued for a particular deployment. It increments whenever a query starts being assigned, and it decrements once a query starts being processed.

However, when a client disconnects before a request is assigned to a replica, the request gets canceled, but the metric never decrements for that request. This causes the metric to increase over time as clients disconnect.

I expect the metric to decrement when a client disconnects and the request is canceled.

Versions / Dependencies

Ray on the latest master.

Reproduction script

# bad_metrics.py

import time
import logging
from ray import serve

logger = logging.getLogger("ray.serve")

@serve.deployment(
    max_concurrent_queries=100,  # 1
    graceful_shutdown_timeout_s=0.0001,
)
def wait(self, *args):
    logger.info("Started a request...")
    time.sleep(100000)
    logger.info("Finished a request...")

app = wait.bind()

Run this app on a Ray cluster that exposes prometheus metrics (e.g. ray start --head --metrics-export-port=8080). Submit a few curl requests from different terminal windows. Then kill the requests. The metric does not decrease.

Issue Severity

Medium: It is a significant difficulty but I can work around it.