[Doc] [KubeRay] Add tutorial for connecting to google cloud storage bucket from GKE RayCluster #38858

architkulkarni · 2023-08-24T23:20:38Z

Why are these changes needed?

This PR adds a self contained tutorial for connecting to a google cloud storage bucket. (Mostly self contained, we do link out to the google cloud docs for creating a bucket.)

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

…nchmark.md Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

…nchmark.md Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

kevin85421 · 2023-08-31T18:18:23Z

doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

+RAY_GCS_BUCKET = "my-bucket"
+RAY_GCS_FILE = "test_file.txt"
+
+ray.init(address="auto")


remote Ray processes can still access the bucket

If we install google-cloud-storage only on the head Pod, will a task scheduled on a worker node still succeed?

doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni · 2023-08-31T19:29:39Z

@kevin85421 Thanks, I addressed the comments

doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni · 2023-08-31T19:42:04Z

Addressed the dependency issue here. 58f81a7

Adds a new YAML file for the GKE bucket tutorial. Resolves ray-project/ray#38858 (comment) --------- Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni · 2023-08-31T22:39:16Z

Test failures:

FAILED bazel/::data-loading-preprocessing.rst
tests:test_multiprocessing
tests:test_task_events_2
tests:test_namespace

These are all unrelated

…ucket from GKE RayCluster (ray-project#38858) This PR adds a self contained tutorial for connecting to a google cloud storage bucket. (Mostly self contained, we do link out to the google cloud docs for creating a bucket.) --------- Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: angelinalg <[email protected]>

…38857) (#39186) * [Doc] [KubeRay] Add tutorial for connecting to google cloud storage bucket from GKE RayCluster (#38858) This PR adds a self contained tutorial for connecting to a google cloud storage bucket. (Mostly self contained, we do link out to the google cloud docs for creating a bucket.) --------- Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: angelinalg <[email protected]> * [Doc] [KubeRay] Add end-to-end tutorial for real-world RayJob workload (batch inference) (#38857) This PR adds a tutorial for running a batch inference workload on KubeRay using the RayJob CRD. It also updates the GPU/GKE doc (which is used as a subroutine in this tutorial) to remove the instructions related to taints and tolerations and GPU driver installation, both of which are currently handled automatically by GKE. --------- Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]> --------- Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: angelinalg <[email protected]>

…ucket from GKE RayCluster (ray-project#38858) This PR adds a self contained tutorial for connecting to a google cloud storage bucket. (Mostly self contained, we do link out to the google cloud docs for creating a bucket.) --------- Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: angelinalg <[email protected]>

…ucket from GKE RayCluster (ray-project#38858) This PR adds a self contained tutorial for connecting to a google cloud storage bucket. (Mostly self contained, we do link out to the google cloud docs for creating a bucket.) --------- Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: angelinalg <[email protected]> Signed-off-by: Jim Thompson <[email protected]>

Adds a new YAML file for the GKE bucket tutorial. Resolves ray-project/ray#38858 (comment) --------- Signed-off-by: Archit Kulkarni <[email protected]>

…ucket from GKE RayCluster (ray-project#38858) This PR adds a self contained tutorial for connecting to a google cloud storage bucket. (Mostly self contained, we do link out to the google cloud docs for creating a bucket.) --------- Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: angelinalg <[email protected]> Signed-off-by: Victor <[email protected]>

kevin85421 and others added 30 commits August 20, 2023 16:58

add docs

d34a528

update

8030e26

add rayservice doc

7f1c300

remove RayService

f2ab31b

update

7225343

update

b92099f

kubeflow doc

9d602ba

update

e5564a8

update

26bc93a

update

7a573d3

update

743f078

update

70928ce

update

e7f4a58

update

0a46948

update

5d2b814

update

702f129

update

dbe0bb5

update

b5b6437

update

8c0dab7

Update doc/source/cluster/kubernetes/benchmarks/memory-scalability-be…

3f43229

…nchmark.md Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Update doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

c8e4fbc

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Update doc/source/cluster/kubernetes/benchmarks/memory-scalability-be…

856d8bc

…nchmark.md Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Update doc/source/cluster/kubernetes/benchmarks/memory-scalability-be…

5b7c2b0

…nchmark.md Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Update doc/source/cluster/kubernetes/benchmarks/memory-scalability-be…

4d86c7c

…nchmark.md Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Update doc/source/cluster/kubernetes/k8s-ecosystem/prometheus-grafana.md

d2df3ed

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Apply suggestions from code review

76831fd

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Apply suggestions from code review

99a97d7

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

Apply suggestions from code review

7bc8d3d

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

update

22730dd

Apply suggestions from code review

2f2794d

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]>

kevin85421 reviewed Aug 31, 2023

View reviewed changes

architkulkarni and others added 10 commits August 31, 2023 12:21

Update doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

77875d1

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]>

Update doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

0cdf429

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]>

Update doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

484e028

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]>

Update doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

f835f3e

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]>

Update doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

ea8e899

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]>

Update doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

7746a44

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]>

Update doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

e163585

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]>

Update doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

9dd9691

Co-authored-by: angelinalg <[email protected]> Signed-off-by: Archit Kulkarni <[email protected]>

rename RAY to GCP and remove shutdown()

cbfaa0c

Merge branch 'master' into gcs-gke-bucket-doc

671ee6c

kevin85421 approved these changes Aug 31, 2023

View reviewed changes

architkulkarni commented Aug 31, 2023

View reviewed changes

doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md Outdated Show resolved Hide resolved

Update doc/source/cluster/kubernetes/user-guides/gke-gcs-bucket.md

58f81a7

Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni closed this in ray-project/kuberay#1372 Aug 31, 2023

architkulkarni added a commit to ray-project/kuberay that referenced this pull request Aug 31, 2023

[Doc] Add gke bucket yaml (#1372)

41018bc

Adds a new YAML file for the GKE bucket tutorial. Resolves ray-project/ray#38858 (comment) --------- Signed-off-by: Archit Kulkarni <[email protected]>

architkulkarni reopened this Aug 31, 2023

architkulkarni added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Aug 31, 2023

architkulkarni merged commit 3034965 into ray-project:master Aug 31, 2023
2 of 3 checks passed

architkulkarni mentioned this pull request Aug 31, 2023

[Doc] [KubeRay] Add two kuberay tutorials (Cherry-pick of #38858 and #38857) #39186

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Doc] [KubeRay] Add tutorial for connecting to google cloud storage bucket from GKE RayCluster #38858

[Doc] [KubeRay] Add tutorial for connecting to google cloud storage bucket from GKE RayCluster #38858

architkulkarni commented Aug 24, 2023 •

edited

Loading

kevin85421 Aug 31, 2023

architkulkarni commented Aug 31, 2023

architkulkarni commented Aug 31, 2023

architkulkarni commented Aug 31, 2023

[Doc] [KubeRay] Add tutorial for connecting to google cloud storage bucket from GKE RayCluster #38858

[Doc] [KubeRay] Add tutorial for connecting to google cloud storage bucket from GKE RayCluster #38858

Conversation

architkulkarni commented Aug 24, 2023 • edited Loading

Why are these changes needed?

Related issue number

Checks

kevin85421 Aug 31, 2023

Choose a reason for hiding this comment

architkulkarni commented Aug 31, 2023

architkulkarni commented Aug 31, 2023

architkulkarni commented Aug 31, 2023

architkulkarni commented Aug 24, 2023 •

edited

Loading