[Post v0.5.0] Remove init containers from YAML files

[kevin85421]

Why are these changes needed?

See #973 for more details.

• KubeRay APIServer

  • apiserver/pkg/util/cluster.go

• Python client

  • use-raw-with-api.py => Follow the instructions in Release v0.5.0 python client library validation #1006
  • test_api.py => CI
  • test_utils.py => CI

• Helm Charts

  • raycluster-cluster.yaml
  • values.yaml

• Sample YAML files

  • RayCluster
    • ray-cluster.autoscaler.yaml => test_sample_raycluster_yamls.py
    • ray-cluster.complete.yaml => test_sample_raycluster_yamls.py
    • ray-cluster.external-redis.yaml => test_sample_raycluster_yamls.py
    • ray-cluster.heterogeneous.yaml => test_sample_raycluster_yamls.py
    • ray-cluster.pod-security.yaml => test_security.py
    • ray-cluster.complete.large.yaml (Skip)
    • ray-cluster.autoscaler.large.yaml (Skip)
  • RayJob
    • ray_v1alpha1_rayjob.yaml
  • RayService
    • ray_v1alpha1_rayservice.yaml => test_sample_rayservice_yamls.py

• Template YAML files for end-to-end tests => KubeRay CI

  • ray-cluster.mini.yaml.template
  • ray-cluster.ray-ft.yaml.template
  • ray-service.yaml.template

test_sample_raycluster_yamls.py

RAY_IMAGE=rayproject/ray:2.3.0 OPERATOR_IMAGE=kuberay/operator:v0.5.0 python3 tests/test_sample_raycluster_yamls.py 2>&1 | tee log

test_sample_rayservice_yamls.py

RAY_IMAGE=rayproject/ray:2.3.0 OPERATOR_IMAGE=kuberay/operator:v0.5.0 python3 tests/test_sample_rayservice_yamls.py 2>&1 | tee log

Test RayCluster Helm chart

kind create cluster --image=kindest/node:v1.23.0
helm install kuberay-operator kuberay/kuberay-operator --version 0.5.0
# path: helm-chart/ray-cluster
helm install raycluster .

test_security.py

# path: kuberay/
OPERATOR_IMAGE=kuberay/operator:v0.5.0 python3 tests/test_security.py

Test RayJob YAML (ray_v1alpha1_rayjob.yaml)

kind create cluster --image=kindest/node:v1.23.0
helm install kuberay-operator kuberay/kuberay-operator --version 0.5.0
# path: ray-operator/config/samples
kubectl apply -f ray_v1alpha1_rayjob.yaml

# Wait for 1 min
kubectl describe rayjobs.ray.io rayjob-sample | grep "Job Status"
# Job Status:             SUCCEEDED

Related issue number

#973
Closes #974

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[kevin85421]

cc @jasoonn

[kevin85421]

@Jeffwan @scarlet25151 @wilsonwang371 would you mind reviewing the changes in apiserver/pkg/util/cluster.go? Thanks!

[kevin85421]

I will update the kuberay-helm repository after this PR is merged.

[scarlet25151]

This PR remove initContainer config in API server, and move the config logic to #973 config is aligned with examples'. /LGTM.

[Jeffwan]

sorry for late. BTW, try to get more background on this. The workers will result in failure at the begining, even it will restart, it will become misleading from observability perspective. Is that a concern?

[kevin85421]

Currently, the old init container logic is wrong. It waits for the head service rather than GCS server. The head service will be ready when the image pull finishes. It will not fail in regular conditions because Ray internally will retry multiple times to connect to the GCS server. However, the retry mechanism in Ray has a timeout, so the workers will fail if the GCS server cannot be ready within the timeout.

In #973, we inject a default init container using ray health-check to wait until the GCS server is ready. We removed the nslookup init container after release v0.5.0 to keep the compatibility philosophy of KubeRay (See #940 for more details).

[skliarpawlo]

Hi, I also have a qq about this change, it seems like it prevents helm chart users to configre docker image to be used for the init container. Is there something I'm missing? We have a strong requirement to use internal docker registry so this makes the chart 0.5.0 not usable as-is for us. I checked that the init container is still created and not removed completely

[kevin85421]

Hi @skliarpawlo, your question is unrelated to KubeRay APIServer, so I open a new thread to answer your question.

[kevin85421]

Hi @skliarpawlo,

In #973, the default init container will be injected by the KubeRay operator automatically, and the injected container will use the same Docker image as the Ray container in the same Pod.

We have a strong requirement to use internal docker registry so this makes the chart 0.5.0 not usable as-is for us. I checked that the init container is still created and not removed completely

This PR is merged after KubeRay v0.5.0 chart is release. Hence, the Helm chart will still have two init containers. I will update the Helm chart release tomorrow. The workaround for you is removing the following lines in your local raycluster-cluster.yaml.

[skliarpawlo]

Hi @skliarpawlo,

In #973, the default init container will be injected by the KubeRay operator automatically, and the injected container will use the same Docker image as the Ray container in the same Pod.

We have a strong requirement to use internal docker registry so this makes the chart 0.5.0 not usable as-is for us. I checked that the init container is still created and not removed completely

This PR is merged after KubeRay v0.5.0 chart is release. Hence, the Helm chart will still have two init containers. I will update the Helm chart release tomorrow. The workaround for you is removing the following lines in your local raycluster-cluster.yaml.

Thank you!
I'm trying to use ray-cluster helm chart as a dependency, not copy it locally to be able to upgrade it easily.
In case the fixed helm chart is released soon, I can wait until than and then just refresh my dependencies.

[kevin85421]

Hi @skliarpawlo, I create release v0.5.1 for Helm charts. You can use:

helm repo update
helm install kuberay-operator kuberay/kuberay-operator --version 0.5.1
helm install raycluster kuberay/ray-cluster --version 0.5.1