[Feature] Connect to RayCluster via GCS port rather than Ray client in compatibility test

[kevin85421]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Based on my observation, the GCS FT bugs (#634 and #638) are easily triggered when we connect to RayCluster via GCS port. Hence, we temporarily connect to RayCluster via Ray client. However, the Ray community plans to de-emphasizes Ray client. Therefore, we need to update ray.init when #634 and #638 are fixed. See #844 (comment) for more details.

Use case

No response

Related issues

#634
#638
#844

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

The flakiness caused by ray.init() is because it shuts down the process with exit code 1 rather than throwing an exception when it fails to connect to the cluster if the cluster is not ready.

[kevin85421]

To clarify, if we use sys.exit(1) to exit the process, it will raise an exception that can be caught by BaseException. However, when the process exits due to QuickExit, no exception is thrown. As a result, the process will terminate after the first call of ray.init() and will not attempt a retry. On the other hand, if the cluster is not ready, Ray client will throw a Python exception.

kuberay/tests/config/ray-cluster.ray-ft.yaml.template

Lines 243 to 253 in 4fbdb9e

	def retry_with_timeout(func, timeout=90):
	err = None
	start = time.time()
	while time.time() - start <= timeout:
	try:
	return func()
	except BaseException as e:
	err = e
	finally:
	time.sleep(1)
	raise err

• Logs for ray.init failure.

  2023-08-26 11:38:30,134	INFO worker.py:1330 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS
  2023-08-26 11:38:30,135	INFO worker.py:1459 -- Connecting to existing Ray cluster at address: 10.244.0.7:6379...
  2023-08-26 11:38:30,203	INFO worker.py:1640 -- Connected to Ray cluster. View the dashboard at �[1m�[32mhttp://10.244.0.7:8265 �[39m�[22m
  [2023-08-26 11:38:39,210 E 244 258] core_worker_process.cc:216: Failed to get the system config from raylet because it is dead. Worker will terminate. Status: GrpcUnavailable: RPC Error message: failed to connect to all addresses; RPC Error details:  .Please see `raylet.out` for more details.
  command terminated with exit code 1
  

Ray may need to have consistent behaviors to thrown exceptions between different methods for connecting with Ray.