[Bug] A mini RayService requires more than 90 seconds to converge

[kevin85421]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

See #837 for more details.

test_sample_rayservice_yamls.py is very flaky (~20% to fail). Previously, it was very stable. #731 said that "I have run more than ten times on my cluster, and the result is always pass.".

• [root cause]: RayServiceAddCREvent fails to converge due to the absence of rayservice-sample-serve-svc in 90 seconds. It is too slow.

Reproduction script

Run the following command several times.

RAY_IMAGE=rayproject/ray:2.1.0 OPERATOR_IMAGE=kuberay/operator:nightly python3 tests/test_sample_rayservice_yamls.py

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

Related slack thread: https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1675703191936259

[architkulkarni]

Just recording the time it took when I ran this test on my laptop:
--- RayServiceAddCREvent 205.55406618118286 seconds ---

[kevin85421]

It becomes stable after #1000. The RayService test passes 25 times consecutively.

[kevin85421]

RAY_IMAGE=rayproject/ray:2.4.0 OPERATOR_IMAGE=kuberay/operator:nightly python3 tests/test_sample_rayservice_yamls.py

The RayServiceAddCREvent converges in 32 seconds. I guessed the root cause of this issue is RAY_IMAGE. The current test framework will not replace the RAY_IMAGE automatically, so the Pod will pull the image from DockerHub instead if RAY_IMAGE is not the same as the value of image in the YAML file.