Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inject cluster name as an environment variable into head and worker pods #934

Merged
merged 5 commits into from
Mar 14, 2023

Conversation

Yicheng-Lu-llll
Copy link
Contributor

@Yicheng-Lu-llll Yicheng-Lu-llll commented Feb 28, 2023

Why are these changes needed?

See #928.

Now user can get env var RAY_CLUSTER_NAME in head and worker pods.

Related issue number

Closes #928

Checks

rootPath={Your kuberay repo path}
# ray-cluster.complete.yaml will create one head pod and one worker pod.
kubectl apply -f $rootPath/ray-operator/config/samples/ray-cluster.complete.yaml
kubectl exec -it $(kubectl get pods -o=name | grep head) -- env | grep RAY_CLUSTER_NAME
kubectl exec -it $(kubectl get pods -o=name | grep worker) -- env | grep RAY_CLUSTER_NAME
# Now change the pod replicas to 2, kuberay operator will create a new worker pod, check if the newly created pod also have env vars.
kubectl apply -f $rootPath/ray-operator/config/samples/ray-cluster.complete.yaml
kubectl exec -it raycluster-complete-worker-small-group-fw6kk -- env | grep RAY_CLUSTER_NAME


# will always see:
# RAY_CLUSTER_NAME=raycluster-complete

Copy link
Member

@kevin85421 kevin85421 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution! Could you

(1) Think about the comparisons between injecting the env variable by KubeRay operator or setting by YAML files.

(2) Add some tests.

(3) Test autoscaler.

ray-operator/controllers/ray/raycluster_controller.go Outdated Show resolved Hide resolved
Copy link
Member

@kevin85421 kevin85421 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check whether BuildPod will always be called after DefaultWorkerPodTemplate and DefaultHeadPodTemplate.

@kevin85421
Copy link
Member

Signed-off-by: Yicheng-Lu-llll <[email protected]>
@Yicheng-Lu-llll
Copy link
Contributor Author

This looks like a better solution?

https://github.com/ray-project/kuberay/blob/master/ray-operator/controllers/ray/common/pod.go#L386

Updated. Thank you for pointing that out!

Copy link
Member

@kevin85421 kevin85421 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a document for environment_variables? Thanks!

ray-operator/controllers/ray/common/pod.go Outdated Show resolved Hide resolved
ray-operator/controllers/ray/common/pod_test.go Outdated Show resolved Hide resolved
ray-operator/controllers/ray/common/pod_test.go Outdated Show resolved Hide resolved
ray-operator/controllers/ray/common/pod_test.go Outdated Show resolved Hide resolved
ray-operator/controllers/ray/common/pod_test.go Outdated Show resolved Hide resolved
Signed-off-by: Yicheng-Lu-llll <[email protected]>
Signed-off-by: Yicheng-Lu-llll <[email protected]>
Copy link
Member

@kevin85421 kevin85421 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also update this line?

and this one

FieldPath: "metadata.labels['ray.io/cluster']",

Signed-off-by: Yicheng-Lu-llll <[email protected]>
Copy link
Member

@kevin85421 kevin85421 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I also tested this PR manually.

Screen Shot 2023-03-13 at 4 53 17 PM

@kevin85421 kevin85421 merged commit 87dde22 into ray-project:master Mar 14, 2023
lowang-bh pushed a commit to lowang-bh/kuberay that referenced this pull request Sep 24, 2023
…ods (ray-project#934)

Inject cluster name as an environment variable into head and worker pods. With this PR, users can use $RAY_CLUSTER_NAME to get RayCluster's name in Pods.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Inject cluster name as an environment variable into head and worker pods
2 participants