[RayService] Failed to update error when head pod is gone

[simon-mo]

First of all, this error handling block is broken, the logger should log errStatus instead of the global err.

kuberay/ray-operator/controllers/ray/rayservice_controller.go

Lines 193 to 196 in 68faa10

	if errStatus := r.Status().Update(ctx, rayServiceInstance); errStatus != nil {
	rayServiceLog.Error(err, "Fail to update status of RayService", "rayServiceInstance", rayServiceInstance)
	return ctrl.Result{}, err
	}

But if we do explicitly log the errStatus message, under one failure condition, it will say:

"error": "Operation cannot be fulfilled on rayservices.ray.io \"rayservice-sample\": the object has been modified; please apply your changes to the latest version and try again", "error": "Operation cannot be fulfilled on rayservices.ray.io \"rayservice-sample\": the object has been modified; please apply your changes to the latest version and try again"}

Steps to repro:

1. create the sample rayservice
2. kubectl delete pod HEAD_POD
3. tail the controller logs

[brucez-anyscale]

Thanks for finding the typo issue. Will fix it soon.
The second issue is quite common, it is due to the resource version of the CR changes, and it will try another reconcile loop.
It is a normal case.

[simon-mo]

@brucez-anyscale but I would like to understand why would the CR change once, and again in here, in the same Reconcile loop iteration?

[brucez-anyscale]

If we do a very simple reconcile loop:
Read a CR, update state, Update the CR. I think the issue still there. It should be a common case in k8s.
If you have more insights or find out we update CR twice in the loop, that could fix the issue!