[Feature] Add liveness and readiness probes for Ray worker pods.

[DmitriGekhtman]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

The KubeRay operator should inject liveness and readiness probes for Ray pods.
Ray provides existing functionality ray health-check that should work for this purpose.
Readiness information should be carried into the RayCluster CR's status field.

The Ray autoscaler's resource heartbeat health-check should be toggled off once this is implemented.
(need to provide a flag in the Ray autoscaler code to turn that function off).

Use case

Better management of Ray worker pods!

Related issues

Exposing status info in the CRD
#223

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[DmitriGekhtman]

@wuisawesome @iycheng @mwtian come to think of it, ray health-check is not a good health check of a Ray worker -- it's actually a remote check of the GCS.

Is there a good way to health check a Ray worker node?

[DmitriGekhtman]

cc @akanso @Jeffwan @wilsonwang371
cc @daikeshi @davidxia re: the connection to RayCluster.Status

[wuisawesome]

oh whoops i missed the sync, but we should implement it as a flag for ray healthcheck.

There's a question of if we should attempt to directly ping the raylet, or ask GCS/get the status from the NodeTable.

Who would be responsible for probing? k8s deployment controller? or kuberay operator?

[DmitriGekhtman]

Who would be responsible for probing? k8s deployment controller? or kuberay operator?

The kubelet of the node on which the Ray pod is scheduled.

There's a question of if we should attempt to directly ping the raylet, or ask GCS/get the status from the NodeTable.

Maybe both? Especially if there's a possibility that these could fail independently.

[DmitriGekhtman]

This is being covered in the context of Ray HA work. Closing to deduplicate.