Subscription Content Access

[adambkaplan]

Meta-enhancement to describe how workloads can access RHEL subscription content in OpenShift 4 clusters. This outlines the key features and components that will allow workloads to have
entitlement keys inejcted. These keys and yum repo configurations are needed to access RHEL
subscription RPMs.

The following components are proposed:

1. A CSI driver to project Secrets and ConfigMaps across namespaces, with appropriate RBAC.
2. A Cluster Subscription Manager to store entitlement keys and yum repo configurations onto the
   cluster.
3. A mutating admission webhook that allows workloads to opt into having entitlement keys injected.

A separate enhancement for OpenShift Builds leveraging these capabilities is also proposed.

[adambkaplan]

/assign @bparees

/cc @gabemontero @coreydaley @otaviof @sbose78 @abhgupta @luciddreamz @derekwaynecarr

[derekwaynecarr]

Noting here so I don’t forget later, but is the csi driver itself only able to read the one named secret?

[bparees]

the CSI driver design is generic, the intent is that it can read any configmap or secret in the cluster, because any of them could be the target of a projection/mount. Auth/admission logic is what prevents a particular user from accessing/mounting a particular secret.

[gabemontero]

If it helps @derekwaynecarr the sample yaml we settled on based on prompting from @deads2k :

kind: ProjectedResource
apiVersion: projected-resource.storage.openshift.io/v1
metadata:
  name: shared-cool-secret
spec:
  backingResource:
    kind: Secret
    apiVersion: v1
    namespace: ns-one
    name: cool-secret
  description: "used to access my internal git repo, RHEL entitlements, or something else cool"
status:
  conditions:
  ...

[adambkaplan]

Furthermore we will need a ConfigMap to mount in the right redhat.repo file (these need to be different if you're running a rhel7 or rhel8 container).

@gabemontero food for thought - k8s supports a ProjectedVolume which lets you combine Secrets and ConfigMaps [1]. We should separately discuss if this is something we want to support in the CSI driver API.

[1] https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#volumeprojection-v1-core

[gabemontero]

interesting @adambkaplan ... yeah add it as a note to the appropriate Jira in this space you recently opened

[derekwaynecarr]

Worth discussing the behavior if the customer is using rhel 7 hosts in a cluster.

Worth discussing how an existing cluster will get the secret when upgrading. Will we alert in absence of secret? What was thinking around that?

[bparees]

i don't think the presence of rhel7 hosts changes this. Yes technically those hosts might have subscriptions we could hostmount, but i don't think we should hostmount them. Even if the pod is running on a rhel7 host, we could just continue using the CSI driver to get the cluster's subscription config/creds into the pods.

[adambkaplan]

This could be interesting, because if a RHEL 7 host is subscribed independently of the cluster, the containers on that host already have access to entitlements. The defaults in cri-o pass those through to containers (via mounts.conf). In this scenario, "do nothing" means your container could consume the host entitlement, but adding the annotation lets you consume a different entitlement with potentially different access.

[adambkaplan]

@derekwaynecarr I added this as a "Risk" - we could address by having the Subscription Injection operator apply a default annotation to every pod.

[adambkaplan]

Updated. However, adding the annotation to every pod introduces a worse risk of pods being denied. The current proposal for the injection operator+webhook will use a custom resource to expose the entitlement, and a SAR checks against the custom resource to ensure the requesting service account has permission to read the entitlement.

Is there an easy way to make a custom resource readable to every service account?

[gabemontero]

Maybe aggregate it to the view role, but the view role is not bound to every SA by default IIRC.

So we are talking an operator that is monitoring namespace creation a la what is done with the builder SA, etc.

At most it would be something a cluster admin would opt into.

Whether that is "easy" ....

[adambkaplan]

Per @dmage - we can address this by granting read permissions to the system:serviceaccount group.

[bparees]

i like the idea of introducing this(the subscription manager, subscription admission webhook, as well as the CSI driver) as a community olm operator first (with no support/expectations). that seems like a good way to bypass all the upgrade/support/tech preview conversations while iterating on it. Not sure if @derekwaynecarr has thoughts on how acceptable that is. (we'd basically abandon the community operator in place once we were ready to release a supported one)

[bparees]

the CSI driver design is generic, the intent is that it can read any configmap or secret in the cluster, because any of them could be the target of a projection/mount. Auth/admission logic is what prevents a particular user from accessing/mounting a particular secret.

[bparees]

i don't think the presence of rhel7 hosts changes this. Yes technically those hosts might have subscriptions we could hostmount, but i don't think we should hostmount them. Even if the pod is running on a rhel7 host, we could just continue using the CSI driver to get the cluster's subscription config/creds into the pods.

[gabemontero]

A few days later than everyone else but I've got some comments/suggestions below. Nothing too major I think.

[gabemontero]

make projected-resource ... a link to https://github.com/openshift/enhancements/blob/master/enhancements/cluster-scope-secret-volumes/csi-driver-host-injections.md

[adambkaplan]

will do on the next push.

[gabemontero]

If it helps @derekwaynecarr the sample yaml we settled on based on prompting from @deads2k :

kind: ProjectedResource
apiVersion: projected-resource.storage.openshift.io/v1
metadata:
  name: shared-cool-secret
spec:
  backingResource:
    kind: Secret
    apiVersion: v1
    namespace: ns-one
    name: cool-secret
  description: "used to access my internal git repo, RHEL entitlements, or something else cool"
status:
  conditions:
  ...

[sbose78]

we'd basically abandon the community operator in place once we were ready to release a supported one

Could this possibly live on as an OLM operator if we end up supporting installation of OLM operators ootb at install time, kinda the "on-top-of.." story ?

[gabemontero]

updates stemming from my last set of comments look good @adambkaplan

[adambkaplan]

Updated to include certificate rotation. Entitlement keys can be revoked and rotated.

[gabemontero]

How does the cluster subscription manager receive the key rotation events @adambkaplan ? Is it the thing that registers the cluster with Red Hat's cloud manager (and the registration includes supplying some sort of callback/listener)? Or does some other component within OCP already doing registering and recording events somewhere?

Some hint on the mechanics could be useful here IMO.

[adambkaplan]

@gabemontero to be honest, I don't know enough w.r.t entitlement rotation/revocation works today to say how this would be implemented. I can add this as an open question - presumably there are existing APIs/infra to do this in RHEL today.

[adambkaplan]

@bparees I think this is ready for final review. Commits have been squashed.

[bparees]

nice job summarizing the high level pieces and how they all interact/come together to provide the overall function.

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adambkaplan, bparees

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [adambkaplan,bparees]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[akostadinov]

Would it be possible for users to specify their own subscription instead of only using Cluster subscription? Cluster might be operating under whatever production subscription organization has. But developers may want to try unavailable channels.