[Bug] Fix flaky sample YAML tests

[kevin85421]

Why are these changes needed?

test-raycluster-sample-yamls-nightly-operator becomes very flaky recently. #1556 introduces a new YAML file for a RayCluster with GCS FT. As a result, the sample YAML tests have become flaky. There are two YAMLs, ray-cluster.external-redis.yaml and ray-cluster.external-redis-uri.yaml, for GCS FT-enabled RayCluster custom resources. If the Redis Pod from the first YAML isn't deleted in time, the RayCluster from the second YAML will connect to the terminating Redis. Consequently, the RayCluster crashes when the Redis Pod is eventually deleted.

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[kevin85421]

cc @rueian

[rueian]

Hi @kevin85421, I am sorry about that.

I thought the whole kind cluster will be recreated before each run of yaml. Wasn’t that the case?

[kevin85421]

Hi @kevin85421, I am sorry about that.

I thought the whole kind cluster will be recreated before each run of yaml. Wasn’t that the case?

No need to apologize. It's quite common, and I also overlooked that part during the code review. I cc'd you primarily because you're planning to improve CI, and I wanted to ensure you have enough context for the change. In compatibility-test.py, the Kind cluster is recreated for each test class, but the sample YAML tests are not.

[architkulkarni]

Out of curiosity can this situation happen in real life outside of CI? (delete one raycluster and start another one very quickly, and the second one fails because it connects to the first one's redis)

[kevin85421]

Out of curiosity can this situation happen in real life outside of CI? (delete one raycluster and start another one very quickly, and the second one fails because it connects to the first one's redis)

Yes, you can try to run both ray-cluster.external-redis.yaml and ray-cluster.external-redis-uri.yaml consecutively.