[Serve] Add serve_deployment_replica_healthy gauge to check whether deployment replicas are healthy

[shrekris-anyscale]

Why are these changes needed?

The Serve controller periodically health checks deployment replicas and restarts them if they're unhealthy. However, this info is only exposed in the controller's logs, which makes it hard to observe.

This change adds a new gauge, serve_deployment_replica_healthy, to track whether deployment replicas are healthy. The gauge is 1 when the replica is healthy and 0 when unhealthy.

Related issue number

N/A

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
    • test_serve_metrics_for_successful_connection in test_metrics.py is updated to check for the new serve_deployment_replica_healthy metric.
  • Manual tests
    • This feature was also manually tested with this Serve app:

from ray import serve

@serve.deployment(
    num_replicas=3,
    ray_actor_options={"num_cpus": 0},
    health_check_period_s=1,
    health_check_timeout_s=1,
)
class A:

    def __init__(self):
        self.tries = 5
    
    def __call__(self, *args):
        import os
        return os.getpid()
    
    def check_health(self):
        self.tries -= 1
        if self.tries < 0:
            raise RuntimeError("Failed!")

app = A.bind()

[simon-mo]

This should work. One alternatively is to count the number of failing healthcheck so it can be aggregated over time (x failing health check in y minutes). Nevertheless I think this is the right start