Kuberay integration #2383

juliusvonkohout · 2023-02-07T14:15:56Z

Which issue is resolved by this Pull Request:
Resolves kubeflow/kubeflow#6680

@kevin85421

Description of your changes:

Checklist:

Unit tests pass:
Make sure you have installed kustomize == 3.2.1
1. make generate-changed-only
2. make test

kevin85421 · 2023-02-16T01:50:37Z

juliusvonkohout · 2023-02-16T09:19:30Z

Does 0.4.0 have the securitycontext set everywhere?

kevin85421 · 2023-02-17T21:20:17Z

Does 0.4.0 have the securitycontext set everywhere?

Yes, the PR ray-project/kuberay#752 also includes in the release 0.4.0. See here for more details.

juliusvonkohout · 2023-02-20T08:43:29Z

Does 0.4.0 have the securitycontext set everywhere?

Yes, the PR ray-project/kuberay#752 also includes in the release 0.4.0. See here for more details.

Are they also enforced everywhere?

juliusvonkohout · 2023-02-27T13:34:18Z

i do not see the securitycontext in raycluster_example.yaml or resources.yaml. Shouldnt there be

          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault

everywhere as shown in https://github.com/ray-project/kuberay/pull/750/files ?

kevin85421 · 2023-02-27T19:44:43Z

i do not see the securitycontext in raycluster_example.yaml or resources.yaml. Shouldnt there be
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault
everywhere as shown in https://github.com/ray-project/kuberay/pull/750/files ?

Good catch. Will update it today.

kevin85421 · 2023-02-28T01:57:59Z

This PR is ready for review cc @juliusvonkohout @richardsliu. Could you approve the CI workflow? Thanks!

cc some KubeRay folks @architkulkarni @DmitriGekhtman @gvspraveen @Yicheng-Lu-llll @mjconnor @Jeffwan

juliusvonkohout · 2023-02-28T11:37:35Z

i do not see the securitycontext in raycluster_example.yaml or resources.yaml. Shouldnt there be
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault
everywhere as shown in https://github.com/ray-project/kuberay/pull/750/files ?
Good catch. Will update it today.

What about the initcontainer securitycontext?
Is there a reason for the logfile in the repository?
We have to check https://github.com/kubeflow/manifests/blob/master/proposals/20220926-contrib-component-guidelines.md#component-requirements for conformance
the test failed

Run cd contrib/ray/
./test.sh
customresourcedefinition.apiextensions.k8s.io/rayclusters.ray.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/rayjobs.ray.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/rayservices.ray.io serverside-applied
serviceaccount/kuberay-operator serverside-applied
role.rbac.authorization.k8s.io/kuberay-operator serverside-applied
clusterrole.rbac.authorization.k8s.io/kuberay-operator serverside-applied
clusterrole.rbac.authorization.k8s.io/rayjob-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/rayjob-viewer-role serverside-applied
clusterrole.rbac.authorization.k8s.io/rayservice-editor-role serverside-applied
clusterrole.rbac.authorization.k8s.io/rayservice-viewer-role serverside-applied
rolebinding.rbac.authorization.k8s.io/kuberay-operator serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/kuberay-operator serverside-applied
service/kuberay-operator serverside-applied
deployment.apps/kuberay-operator serverside-applied
deployment.apps/kuberay-operator condition met
NAME                                READY   STATUS    RESTARTS   AGE
kuberay-operator-5b8cd69758-nsvlv   1/1     Running   0          18s
raycluster.ray.io/kubeflow-raycluster created
pod/kubeflow-raycluster-head-z44k6 condition met
error: timed out waiting for the condition on pods/kubeflow-raycluster-worker-small-group-sfxch
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
make: *** [Makefile:11: test] Error 2
Error: Process completed with exit code 2.

kevin85421 · 2023-02-28T15:23:39Z

What about the initcontainer securitycontext?

Added. Btw, what's the reason to add securityContext here? The namespaces created by Kubeflow (example/) do not apply PSS. Why do we set the securityContext by default rather than making users configure that?

Is there a reason for the logfile in the repository?

Removed.

We have to check https://github.com/kubeflow/manifests/blob/master/proposals/20220926-contrib-component-guidelines.md#component-requirements for conformance

I have already checked the following:

README.md
There must be an OWNERS file with at least 2 users
- We have the file OWNERS, but the file seems to not include "users" (@juliusvonkohout)
The component must work with the latest version of Kubeflow, and its dependencies
- Maybe you can do that part. You are more familiar with Kubeflow. (@juliusvonkohout)
There must be an UPGRADE.md file that documents any instructions users need to follow when applying manifests of a newer version
There needs to be sufficient work on testing
- Test exists, but we need to make sure it passes on CI.
- I opened a PR to my fork, and the CI workflow can be triggered without approval. It passes (link).

the test failed

Does KubeRay use free plan GitHub Actions? If so, the CPU may be insufficient. I tried to reduce the CPU request.

[UPDATE] It passes after I reduce the CPU request (link).

Others

I fetched your master branch and rebase it. Everything goes wrong, so I need to force push to make the branch back. Would you mind adding your comments again?
I prefer to use Fig. 1 compared with Fig. 2. The example in README primarily focuses on Jupyter Notebook / Kubeflow Central Dashboard / KubeRay operator. Hence, Fig. 2 includes a lot of information that is unrelated to the example. This will make users confused.

Fig. 1

Fig. 2

Any thoughts? Thanks!

juliusvonkohout · 2023-03-01T13:47:00Z

"Added. Btw, what's the reason to add securityContext here? The namespaces created by Kubeflow (example/) do not apply PSS. Why do we set the securityContext by default rather than making users configure that?" Setting and enforcing are two different things. We want to at least set the securitycontext

juliusvonkohout · 2023-03-01T13:51:17Z

Figure 1 will be wrong quite soon. We rename everything to workbenches. Jupyter notebooks is also wrong it is a Notebook inside a Jupyterlab like a car and gasoline, file and server. Furthermore we also want to call it from KFP directly. We can include both images, but then please use the SVG version and rename to workbench/Jupyterlab, get rid of the notebook term and add KFP.

juliusvonkohout · 2023-03-01T13:52:57Z

And thanks for your amazing effort so far

kevin85421 · 2023-03-01T21:43:21Z

Setting and enforcing are two different things. We want to at least set the securitycontext

Got it. Although the configuration has no relationship with this integration, it is nice to have.

Figure 1 will be wrong quite soon. We rename everything to workbenches.

I will add some sentences to explain it, but I prefer not to change the name of "Kubeflow Central Dashboard" in Figure 1. The example is based on release 1.6, and if we introduce new terms from future releases, users may feel confused. You can open a new PR to update it if you want the example to support the future release.

Add some sentences to explain it.

Jupyter notebooks is also wrong it is a Notebook inside a Jupyterlab like a car and gasoline, file and server.

Fixed.

Furthermore we also want to call it from KFP directly. We can include both images, but then please use the SVG version and rename to workbench/Jupyterlab, get rid of the notebook term and add KFP.

KFP (Kubeflow pipeline) seems not to be a necessary component for users to understand this example. It may make users feel confused. Again, you can open a new PR later to add a section to describe how to launch a JupyterLab instance via KFP.

please use the SVG version

Done

kevin85421 · 2023-03-27T06:09:56Z

Add KFP to the diagram

Done

Do you see Istio initcontainers and sidecars (proxy) for ray head, worker etc?

No, maybe I can try to put the Istio integration as an item for next quarter.

juliusvonkohout · 2023-03-27T18:34:35Z

No, maybe I can try to put the Istio integration as an item for next quarter.

This might be problematic, because in a full Kubeflow they are injected by default.
So why are they not injected in your installation?

https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/ is the documentation

since profile namespaces have the istio-injection=enabled label they will be added by default.

Or did you annotate the ray pods with sidecar.istio.io/inject="false" ?

Or are you not in a Kubeflow profile namespace?

juliusvonkohout · 2023-03-27T18:35:56Z

NAMESPACE=default definitely looks wrong. We need to test in a namespace where also your Jupyterlabs from Kubeflow are running

juliusvonkohout · 2023-04-04T07:38:38Z

@kimwnasptd please approve. The istio enablement is for a second iteration.

juliusvonkohout · 2023-04-04T07:41:00Z

Istio support for ray is tracked here ray-project/kuberay#1005

juliusvonkohout · 2023-04-04T12:01:24Z

@kevin85421 after #2432 i can approve

kimwnasptd · 2023-05-12T16:03:19Z

Thanks for the great work @juliusvonkohout! Awesome to see Ray start being part of Kubeflow

/lgtm
/approve

google-oss-prow · 2023-05-12T16:03:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout, kimwnasptd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kimwnasptd]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow bot added the do-not-merge/work-in-progress label Feb 7, 2023

google-oss-prow bot requested review from PatrickXYS and yanniszark February 7, 2023 14:16

google-oss-prow bot added size/XS size/S and removed size/XS labels Feb 7, 2023

google-oss-prow bot added size/XXL and removed size/S labels Feb 16, 2023

kevin85421 force-pushed the patch-18 branch from 4a38bb9 to c0a6001 Compare February 28, 2023 14:23

ray integration

38e4643

kevin85421 force-pushed the patch-18 branch from c0a6001 to 38e4643 Compare February 28, 2023 14:46

init securityContext

580cda4

kevin85421 added 2 commits February 28, 2023 15:35

reduce cpu request

63ee950

UPGRADE.md

d404cb0

This was referenced Feb 28, 2023

[Docker] (Kubeflow integration) Only UID=1000 has the write access of /home/ray in Ray images ray-project/ray#30959

Closed

Ray restricted podsecuritystandards for enterprise security and Kubeflow integration ray-project/ray#29665

Closed

replace Jupyter notebook with JupyterLab

ab769a8

add KFP to the figure

03d411f

juliusvonkohout added 11 commits April 3, 2023 14:24

Disable istio sidecars for ray head in raycluster_example.yaml

62f05c2

Update README.md

f6163bc

Never ever use the default namespace

c363688

Update README.md

f89f4fc

Disable the ray worker sidecar

7592855

Update kustomization.yaml

6d81964

Create namespace.yaml

1d9c500

Update test.sh to use the right namespace

619e026

Merge branch 'kubeflow:master' into patch-18

20007dd

Update README.md

1550e8a

Update README.md for Kubeflow 1.7

cf3abdd

kevin85421 mentioned this pull request Apr 3, 2023

[Feature] Support Istio ray-project/kuberay#1005

Closed

2 tasks

juliusvonkohout mentioned this pull request Apr 4, 2023

Promote Julius von Kohout from reviewer to approver for /contrib #2432

Merged

1 task

juliusvonkohout changed the title ~~WIP: Kuberay integration~~ Kuberay integration Apr 28, 2023

google-oss-prow bot removed the do-not-merge/work-in-progress label Apr 28, 2023

kimwnasptd mentioned this pull request May 9, 2023

Manifests WG Roadmap for KF 1.8 #2456

Closed

google-oss-prow bot assigned kimwnasptd May 12, 2023

google-oss-prow bot added the lgtm label May 12, 2023

google-oss-prow bot added the approved label May 12, 2023

google-oss-prow bot merged commit afc6a0e into kubeflow:master May 12, 2023

juliusvonkohout mentioned this pull request Sep 15, 2023

Add juliusvonkohout to wg-manifests-leads group #2337

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kuberay integration #2383

Kuberay integration #2383

juliusvonkohout commented Feb 7, 2023

kevin85421 commented Feb 16, 2023 •

edited

Loading

juliusvonkohout commented Feb 16, 2023

kevin85421 commented Feb 17, 2023

juliusvonkohout commented Feb 20, 2023

juliusvonkohout commented Feb 27, 2023

kevin85421 commented Feb 27, 2023

kevin85421 commented Feb 28, 2023

juliusvonkohout commented Feb 28, 2023 •

edited

Loading

kevin85421 commented Feb 28, 2023 •

edited

Loading

juliusvonkohout commented Mar 1, 2023

juliusvonkohout commented Mar 1, 2023

juliusvonkohout commented Mar 1, 2023

kevin85421 commented Mar 1, 2023 •

edited

Loading

kevin85421 commented Mar 27, 2023

juliusvonkohout commented Mar 27, 2023

juliusvonkohout commented Mar 27, 2023 •

edited

Loading

juliusvonkohout commented Apr 4, 2023 •

edited

Loading

juliusvonkohout commented Apr 4, 2023

juliusvonkohout commented Apr 4, 2023 •

edited

Loading

kimwnasptd commented May 12, 2023

google-oss-prow bot commented May 12, 2023

Kuberay integration #2383

Kuberay integration #2383

Conversation

juliusvonkohout commented Feb 7, 2023

kevin85421 commented Feb 16, 2023 • edited Loading

Progress

Screenshot

juliusvonkohout commented Feb 16, 2023

kevin85421 commented Feb 17, 2023

juliusvonkohout commented Feb 20, 2023

juliusvonkohout commented Feb 27, 2023

kevin85421 commented Feb 27, 2023

kevin85421 commented Feb 28, 2023

juliusvonkohout commented Feb 28, 2023 • edited Loading

kevin85421 commented Feb 28, 2023 • edited Loading

Others

juliusvonkohout commented Mar 1, 2023

juliusvonkohout commented Mar 1, 2023

juliusvonkohout commented Mar 1, 2023

kevin85421 commented Mar 1, 2023 • edited Loading

kevin85421 commented Mar 27, 2023

juliusvonkohout commented Mar 27, 2023

juliusvonkohout commented Mar 27, 2023 • edited Loading

juliusvonkohout commented Apr 4, 2023 • edited Loading

juliusvonkohout commented Apr 4, 2023

juliusvonkohout commented Apr 4, 2023 • edited Loading

kimwnasptd commented May 12, 2023

google-oss-prow bot commented May 12, 2023

kevin85421 commented Feb 16, 2023 •

edited

Loading

juliusvonkohout commented Feb 28, 2023 •

edited

Loading

kevin85421 commented Feb 28, 2023 •

edited

Loading

kevin85421 commented Mar 1, 2023 •

edited

Loading

juliusvonkohout commented Mar 27, 2023 •

edited

Loading

juliusvonkohout commented Apr 4, 2023 •

edited

Loading

juliusvonkohout commented Apr 4, 2023 •

edited

Loading