SCC Error on OpenShift

[christianh814]

Deploying the operator using the kustomize method...

$ kubectl apply -k http://github.com/minio/operator
namespace/minio-operator created
customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io created
serviceaccount/minio-operator created
clusterrole.rbac.authorization.k8s.io/minio-operator-role created
clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding created
service/operator created
deployment.apps/minio-operator created

Produces the following output on OpenShift 4.5

$ kubectl get events -n minio-operator
LAST SEEN   TYPE      REASON              OBJECT                                 MESSAGE
110s        Warning   FailedCreate        replicaset/minio-operator-6d6cf76d4c   Error creating: pods "minio-operator-6d6cf76d4c-" is forbidden: unable to validate against any security context constraint: [spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000570000, 1000579999]]
5m56s       Normal    ScalingReplicaSet   deployment/minio-operator              Scaled up replica set minio-operator-6d6cf76d4c to 1

Expected Behavior

In other version of the operator, no scc edits were required. It was as simple as running...

$ kubectl apply -f https://raw.githubusercontent.com/minio/minio-operator/master/minio-operator.yaml

Current Behavior

Now, you need to run this to make it work..

$ kubectl apply -k http://github.com/minio/operator
$ oc adm policy add-scc-to-user anyuid -z minio-operator -n minio-operator
$ oc delete deployment minio-operator -n minio-operator
$ kubectl apply -k http://github.com/minio/operator

This will make the pod run...

$ kubectl get pods -n minio-operator
NAME                              READY   STATUS    RESTARTS   AGE
minio-operator-6d6cf76d4c-b86rd   1/1     Running   0          5m51s

Possible Solution

Any reason why 1000 is used? How was this operator configured before? I never needed to change the SCC definition

Steps to Reproduce (for bugs)

1. Install OpenShift 4.5+ openshift-install create cluster
2. Try and deploy the operator kubectl apply -k http://github.com/minio/operator
3. Observe the failure kubectl get events -n minio-operator

Context

Trying to automate the installation of Minio in my pipeline

Regression

I believe this is a regression since an SCC edit wasn't required before

Your Environment

• Version used (minio-operator): Latest
• Environment name and version (e.g. kubernetes v1.17.2):

$ oc version
Client Version: 4.5.8
Server Version: 4.5.8
Kubernetes Version: v1.18.3+6c42de8

• Operating System and version (uname -a): RHCOS
• Link to your deployment file: None

[harshavardhana]

this is because we choose the security context in our yamls - not sure why Openshift denies this, they should simply honor the security contexts specified in the yaml.

[dvaldivia]

Pod security context was added recently, this was added because we don't need to run as root even inside the container, perhaps the user id chosen is not compatible with openshift as the error suggests?

You could get around this meanwhile by doing an overlay with the fixes that you are doing manually and install the operator in that manner.

Before we had a long long yaml with everything in it, the new kustomize approach is much organized, what changed and broke your install path was the pod security context

[christianh814]

It seems like the 1000 value is out of the allowed range

 Invalid value: 1000: must be in the ranges: [1000570000, 1000579999]]

[dvaldivia]

that error seems to be openshift specific, the kubernetes documentation uses 1000 as an example

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

[dvaldivia]

similar in openshift 4.5 documentation, is there something custom about your setup @christianh814 ?

https://docs.openshift.com/container-platform/4.5/authentication/managing-security-context-constraints.html

[harshavardhana]

@dvaldivia this is part of OpenShift default security context constraints, there is nothing we need to do,

@christianh814 You just have to configure the correct runAsUser or runAsGroup or remove them so that Openshift can force the container to run under a specific SCC

Closing this as not our problem.

[dvaldivia]

@harshavardhana agree, even the documentation for openshift 4.5 shows an example using user 1000 so it may be something on their setup

[smooti]

I figured I would add to this since I have also run into this.

As mentioned by harshavardhana this is an SCC issue when deploying on OpenShift. In order to get the helm chart to deploy to an OpenShift cluster you just need to make sure that the helm chart passes the SCC checks.

For example by default (At least for me running on OpenShift 4.16.7) the user UID must be in-between [1000730000 - 1000739999]. The MinIO helm chart by default has a values.yaml file with pre-defined values including the SCC specifications. You can overwrite these values by specifying a separate operator-values.yaml and passing it into the helm command like so...
helm install --namespace minio-operator --create-namespace operator minio-operator/operator --values operator-values.yaml

The content of my operator-values.yaml...

# Needed in order to pass OpenShift security context constraints

operator:
  securityContext:
    runAsUser: 1000730000
    runAsGroup: 1000730000
    fsGroup: 1000730000

  containerSecurityContext:
    runAsUser: 1000730000
    runAsGroup: 1000730000

[ramondeklein]

@smooti The problem with these IDs on Openshift is that the valid ranges vary per project (source), so it's not possible to assign a default that will just work (AFAIK).

[smooti]

I found great documentation from rook.io in regards to OpenShift and SecurityContextConstraints. It helped me in understanding how they deploy their operator and get a better understanding of OpenShift. Looks like the best thing to do would be to define a SecurityContextConstraint with the minimal permissions the pod needs in order to run within it's own Namespace.
Here is a link to the document and hopefully this helps everybody.

[smooti]

I created a feature request to see if this could be implemented into the operator helm chart #2315