[Feature] Allow wait-gcs-ready to choose or bypass volumes to mount

[richardsliu]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently Kuberay worker pods have an init-container for connecting to the Ray head. It attempts to mount all the volumes that the Ray container mounts: https://github.com/ray-project/kuberay/blob/master/ray-operator/controllers/ray/common/pod.go#L236

It would be helpful to add a way to select which volumes to mount, or bypass mounting completely.

The reason is that some volumes may be backed by other volume types that require special drivers to be installed. For example, GoogleCloudPlatform/gcs-fuse-csi-driver#38.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

You can refer to this document to disable the init container injection and define your own init container.

[richardsliu]

Thanks is there an example of a custom init container?

[kevin85421]

You can refer to pod.go#L200 in #973. I will add a section in FAQ.md for more details later.

[richardsliu]

Also is this approach generally recommended? The comment in the helm chart template warns explicitly against it: https://github.com/ray-project/kuberay/blob/master/helm-chart/kuberay-operator/values.yaml#L85

For the long term, I suspect having user-injected custom init containers may not be sustainable since the code might diverge.

Would it be possible to perhaps extract the actual code injected for the init container to a config map? Then if the user wishes to inject their own logic (like bypassing volumes), they can at least reuse the same container code.

[richardsliu]

Additionally, how about adding some template logic here to allow users to inject their init containers through Helm charts?

[kevin85421]

Also is this approach generally recommended? The comment in the helm chart template warns explicitly against it: https://github.com/ray-project/kuberay/blob/master/helm-chart/kuberay-operator/values.yaml#L85

The job of the init container is quite straightforward: it merely waits for the GCS on the Ray head Pod to be ready. We've added the comment Warning: we highly recommend setting this to true and allowing KubeRay to handle it for you. This is because KubeRay users include not only infrastructure engineers who are familiar with Kubernetes, but also AI practitioners from small startups who focus on ML and may not have experience with Kubernetes.

We include these comments to encourage AI practitioners to use the default configurations and prevent potential misconfigurations. For infrastructure engineers, it's safe to define the init container independently.