[Feature] Ray restricted podsecuritystandards for enterprise security and Kubeflow integration

[kevin85421]

Why are these changes needed?

Kubernetes defines three different Pod Security Standards, including privileged, baseline, and restricted, to broadly
cover the security spectrum. The privileged standard allows users to do known privilege escalations, and thus it is not
safe enough for security-critical applications.

This PR describes how to configure RayCluster YAML file to apply restricted Pod security standard. The following
references can help you understand this document better:

• Kubernetes - Pod Security Standards
• Kubernetes - Pod Security Admission
• Kubernetes - Auditing
• KinD - Auditing

Related issue number

ray-project/ray#29665

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

Screenshot for Step 5.2

Followup issues

• Set default securityContext in KubeRay operator (X)
• Add a test in CI to make sure KubeRay works as expected with the restricted security standard.
• Configure securityContext properly on Pods created by Autoscaler. (Done: exposed related interfaces)
• Configure securityContext properly on Pods specified by RayCluster YAML file. (X)

[juliusvonkohout]

This documentation looks very good. Two things from are missing so far.

1. @DmitriGekhtman said that that there is a dynamic auto scaler container in some environments that is attached to the head pod and must be checked too

2. Not only the cluster but also the operator should be installed in the pod-security namespace and have a proper security context

[juliusvonkohout]

We also checked that kuberay properly has its own serviceaccount https://github.com/ray-project/kuberay/tree/master/ray-operator/config/rbac

[kevin85421]

cc @DmitriGekhtman @juliusvonkohout I updated the PR. Can you help me review this PR? Thank you!

[DmitriGekhtman]

I've exposed the (optional) autoscaler sidecar's security context configuration in this PR:
#752

In a separate PR, we should also tweak the Ray cluster Helm chart to expose Ray containers' securityContext fields.

[DmitriGekhtman]

It looks good to me.

[Jeffwan]

Make sense and looks good to me

[juliusvonkohout]

Looks good so far. I will try to test it tomorrow. One thing you might should test is installing additional python dependencies in the head and workers during runtime. Just to be sure that the file system permissions are all right and that you cann really roll this out to all users.

[DmitriGekhtman]

One thing you might should test is installing additional python dependencies in the head and workers during runtime.

@juliusvonkohout could you clarify how the dependencies should be installed and what behavior you would expect?

[juliusvonkohout]

One thing you might should test is installing additional python dependencies in the head and workers during runtime.

@juliusvonkohout could you clarify how the dependencies should be installed and what behavior you would expect?

I would just like to see pip install tested as described here https://docs.ray.io/en/latest/ray-core/handling-dependencies.html

[DmitriGekhtman]

Got it, we can verify that a runtime env with a pip dependency works!

[juliusvonkohout]

I just tried

Step 4: Install the KubeRay operator

# Path: helm-chart/kuberay-operator
helm install -n pod-security kuberay-operator .

on an openshift cluster. We might need to remove the seccomp stuff or we will get an error.

Error creating: pods "kuberay-operator-5d7d9fc944-h7fcr" is forbidden: unable to validate against any security context constraint: [pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/kuberay-operator: Forbidden: seccomp may not be set
...

So i removed

            seccompProfile:
              type: RuntimeDefault

everywhere

Sadly you are using dockerhub so i will have to wait a day

"Failed to pull image "kuberay/operator:nightly": rpc error: code = Unknown desc = reading manifest nightly in docker.io/kuberay/operator: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"

With a mirror at mtr.devops.telekom.de/juliusvonkohout/kuberay:nightly i at least got the kuberay operator nightly image. I definitely do not recommend to use dockerhub. There is quay.io and other alternatives which do not have those problems. But that is not relevant for this PR.

[DmitriGekhtman]

I definitely do not recommend to use dockerhub. There is quay.io and other alternatives which do not have those problems. But that is not relevant for this PR.

Thanks for the suggestion @juliusvonkohout! Would you mind opening an issue discussing that?

[kevin85421]

Thank @juliusvonkohout for your review!

on an openshift cluster. We might need to remove the seccomp stuff or we will get an error.

Error creating: pods "kuberay-operator-5d7d9fc944-h7fcr" is forbidden: unable to validate against any security context constraint: [pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/kuberay-operator: Forbidden: seccomp may not be set
...

So I removed seccompProfile everywhere

1. By the restricted Pod security standard, spec.securityContext.seccompProfile is necessary. Without seccompProfile, the following error message will be reported.

   seccompProfile (pod or container \"ray-head\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")
   

2. I do not have any experience related to OpenShift. In my understanding, OpenShift plays a resource orchestrator role which is very similar to Kubernetes. What's the relationship between OpenShift and KubeFlow support? Thank you!

[juliusvonkohout]

Alright then this will have to be fixed on the openshift side. They will also switch to PSS soon. Our main target is a normal upstream kubernetes such as kind. I have updated my post. You can ignore the seccomp part then.

and on openshift the image is executed with a random non-root uid. There are filesystem permission errors

$ python3 samples/xgboost_example.py
2022-11-28 14:16:26,296 INFO worker.py:1224 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS
2022-11-28 14:16:26,296 INFO worker.py:1333 -- Connecting to existing Ray cluster at address: 10.131.0.155:6379...
2022-11-28 14:16:26,303 INFO worker.py:1515 -- Connected to Ray cluster. View the dashboard at http://10.131.0.155:8265 
Traceback (most recent call last):
  File "samples/xgboost_example.py", line 28, in <module>
    result = trainer.fit()
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/train/base_trainer.py", line 343, in fit
    tuner = Tuner(trainable=trainable, run_config=self.run_config)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/tuner.py", line 144, in __init__
    self._local_tuner = TunerInternal(**kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/impl/tuner_internal.py", line 104, in __init__
    self._run_config
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/tune/impl/tuner_internal.py", line 261, in _setup_create_experiment_checkpoint_dir
    os.makedirs(path)
  File "/home/ray/anaconda3/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/home/ray/anaconda3/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/ray/ray_results'
$ id
uid=1001190000(1001190000) gid=0(root) groups=0(root),1001190000

you need to chmod 777 some directories.

so the following has to be done

1. We will ignore the seccomp stuff (only applicalbe to openshift) and wiat until it switches to PSS too https://connect.redhat.com/en/blog/important-openshift-changes-pod-security-standards
2. fix the filesystem permissions for docker.io/rayproject/ray-ml:2.0.0 and make sure that your image really runs with any uid. The other option is to add runAsUser: 1000. So either make sure to specify the right user or make sure to be able to run under any userid.
3. Please provide a workload with pip installations and virtualenvs. I was able to just to pip install seaborn when i was running as user 1000 in the head and worker pod, so it should be trivial.

here is the approach without fixing the filesystem properly and doing the runasuser workaround instead.
ray-cluster.pod-security.txt
values.txt

I strongly suggest to properly fix the filesystem instead of encorcing a particular user. These commands will be helpfull for you

RUN mkdir -p ${APPLICATION_ROOT} && chown -R 1000:0 ${APPLICATION_ROOT} && chmod -R 0777 ${APPLICATION_ROOT}
COPY --chown=1000:0 --chmod=777 app-root ${APPLICATION_ROOT}

THis also applies for whatever you do in the busybox initcontainer. If possible just use /tmp and you wont have any trouble with permissions.

[kevin85421]

@juliusvonkohout Do you have the time to chat the details on Zoom? If so, we can communicate via Kubeflow slack workspace. Thank you!

[juliusvonkohout]

And this will fix the seccomp issue https://kubernetes.io/blog/2021/08/25/seccomp-default/ for everyone in the long term. Nothing we have to worry about now.

[kevin85421]

Resolved the conflict and opened an issue for UID in ray-project/ray#30959.

[DmitriGekhtman]

Okidoke, looks good to merge! Thanks.