[Feature][Docs] Explain how to specify container command for head pod

[kevin85421]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Some users want to run additional commands when RayClusters are started. For example, a user wants to run ray start && python3 something.py when the cluster is started.

Users can use headGroupSpec.template.spec.command to set the command running in the head pod. However, there is a tricky behavior that may cause users to be confused. If headGroupSpec.template.spec.command does not contain the substring ray start, KubeRay will replace the command specified by users. See the following code snippet for more details.

kuberay/ray-operator/controllers/ray/common/pod.go

Lines 307 to 325 in e77b095

	if !strings.Contains(cmd, "ray start") {
	cont := concatenateContainerCommand(rayNodeType, rayStartParams, pod.Spec.Containers[rayContainerIndex].Resources)
	// replacing the old command
	pod.Spec.Containers[rayContainerIndex].Command = []string{"/bin/bash", "-lc", "--"}
	if cmd != "" {
	// If 'ray start' has --block specified, commands after it will not get executed.
	// so we need to put cmd before cont.
	args = fmt.Sprintf("%s && %s", cmd, cont)
	} else {
	args = cont
	}
	if !isRayStartWithBlock(rayStartParams) {
	// sleep infinity is used to keep the pod `running` after the last command exits, and not go into `completed` state
	args = args + " && sleep infinity"
	}
	pod.Spec.Containers[rayContainerIndex].Args = []string{args}
	}

This software design will introduce more "unknown unknowns" for users, and thus we need:
(1) Update the document (this issue)
(2) Discuss whether remove this behavior and find its replacement.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[kevin85421]

cc @DmitriGekhtman

[tgaddair]

@kevin85421 I wonder if this could be added in a more explicit way. The current behavior of command is confusing, as it changes based on the presence or absence of a substring, and if the command is overwritten, the rayStartParams essentially become useless (and the user needs to implement this logic themselves).

Maybe there's room for something alongside the rayStartParams like a postStartCommand (which would be invalid if block is set) or similar?

[DmitriGekhtman]

There's some discussion about how to get something approximating the desired behavior with a post-start hook:
https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1669986058864489?thread_ts=1669647595.429959&cid=C02GFQ82JPM

[kevin85421]

@kevin85421 I wonder if this could be added in a more explicit way. The current behavior of command is confusing, as it changes based on the presence or absence of a substring, and if the command is overwritten, the rayStartParams essentially become useless (and the user needs to implement this logic themselves).

Maybe there's room for something alongside the rayStartParams like a postStartCommand (which would be invalid if block is set) or similar?

I totally agreed with your arguments. In addition, I found a post-start hook (commands execute when the RayCluster is READY) is very important recently. The hook can combine both RayCluster creation and other commands as an atomic operation, and thus make KubeRay more idempotent. One of the use cases is to avoid #756.

cc @architkulkarni

[kevin85421]

Add this issue to v0.5.0 release.

[DmitriGekhtman]

Yeah, moving the job submission into the entry point makes sense to me. I don't really see the point of decoupling cluster creation and job submission...
Except that it could be nice to submit a new job to an already running cluster...
But that doesn't work so well anyway due to lack of clean isolation between subsequent jobs.

[kevin85421]

Related discussion: https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1675378764037199

[kevin85421]

Related discussion: https://ray-distributed.slack.com/archives/C02GFQ82JPM/p1676506490994229

Upgrade the priority to P0.