-
Notifications
You must be signed in to change notification settings - Fork 471
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
build: Support Mounted Resource Volumes
Proposal to support mounts for Secrets, ConfigMaps, and ephemeral CSI volumes in builds. This augments prior work which let Secrets and ConfigMaps be used as sources in builds. Unlike the current approach, this enhancement leverages buildah's volume mount feature to let content be available only at build time, and not at runtime. This is useful for builds that need to access private artifact repositories or RHEL subscription content. This current proposal builds on top of Ben Parees's original volume mount proposal. It adds ephemeral `CSI` volumes as a supported volume type, which would make it easier for builds to mount in simple content access certs via the Projected Resource CSI driver. Supported volume types will be gated at the API level to ensure the volume type does not pose a security risk and is correctly lifecycled.
- Loading branch information
1 parent
ec37598
commit be914da
Showing
1 changed file
with
297 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,297 @@ | ||
| --- | ||
| title: volume-mounted-resources | ||
| authors: | ||
| - @bparees | ||
| - @adambkaplan | ||
| reviewers: | ||
| - @derekwaynecarr | ||
| - @smarterclayton | ||
| approvers: | ||
| - @bparees | ||
| creation-date: 2020-01-08 | ||
| last-updated: 2021-04-15 | ||
| status: implementable | ||
| --- | ||
|
|
||
| # Volume Mounted Resources | ||
|
|
||
| ## Release Signoff Checklist | ||
|
|
||
| - [x] Enhancement is `implementable` | ||
| - [x] Design details are appropriately documented from clear requirements | ||
| - [x] Test plan is defined | ||
| - [x] Graduation criteria for dev preview, tech preview, GA | ||
| - [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/) | ||
|
|
||
| ## Open Questions [optional] | ||
|
|
||
| 1. Can we make this the default or even only behavior for builds? | ||
| No, need to make it opt-in to avoid potentially breaking existing buildconfigs. | ||
| 2. What happens if a user overrides the default volume mounts used by Builds today? | ||
| We override the default with the sources chosen in the BuildConfig volumes array. | ||
|
|
||
| ## Summary | ||
|
|
||
| Today builds support getting source input from configmaps and secrets, hereafter referred to as "resources". | ||
| When users utilize this feature, the resource is volume-mounted into the build pod and placed in the "build context" within the build's execution environment, alongside other build sources like git source code. | ||
| The next steps depend on whether it is an s2i or dockerfile build. | ||
|
|
||
| For s2i builds, the generated Dockerfile contains commands to `ADD` the content at a path specified by the user, the assemble script is invoked, and then the injected content is zeroed out prior to committing the image via a `RUN rm` command added to the Dockerfile. | ||
|
|
||
| For dockerfile builds, the user is instructed to add appropriate `ADD` and `RUN rm` commands to their dockerfile to inject the content that is available in the build's working directory (along with their application source, where applicable). | ||
|
|
||
| There are a few undesirable aspects to this: | ||
|
|
||
| 1. In the dockerfile case, the content can still be found in lower layers of the image unless a layer squashing option is selected. | ||
| 2. Requires extra work by the user in the Dockerfile, so each Dockerfile must be customized | ||
|
|
||
| This enhancement proposes to introduce an option to use buildah's capability to mount a volume at build time. | ||
| The content mounted into the build pod would be then mounted into the container processing the Dockerfile, making that content available within the container so Dockerfile commands could reference it. | ||
| No explicit `ADD` would be required, and since mounted content is not committed to the resulting image, no `RUN rm` equivalent is required to clean up the injected content. | ||
|
|
||
| To avoid security and lifecycle concerns, the following volume types will be supported initially: | ||
|
|
||
| 1. Secrets | ||
| 2. ConfigMaps | ||
| 3. `csi` - this is to enable use of [Projected Resource CSI driver](/enhancements/cluster-scope-secret-volumes/csi-driver-host-injections.md) volumes. | ||
|
|
||
|
|
||
| ## Motivation | ||
|
|
||
| ### Goals | ||
|
|
||
| * Simplify how users consume secret + configmap content in builds | ||
| * Increase the security of protected content being injected to images | ||
| * Simplify use cases that require consuming credentials during the build, but need to ensure those credentials do not end up in the output image. | ||
| * Eventually extend this api to allow the mounting of traditional volumes (such as those backed by persistent storage) | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| * This enhancement should not result in a change of behavior for users of the existing secret/configmap injection api. | ||
| * Provide immediate support for persistent volume claims in builds. | ||
| This is a long term goal that will be addressed in a future enhancement proposal. | ||
|
|
||
|
|
||
| ## Proposal | ||
|
|
||
| ### User Stories [optional] | ||
|
|
||
| The enabled use cases are essentially identical to what can be done with the configmap/secret input api in builds today, but with a better user experience and security as discussed above | ||
| It does not enable a new use case that is not already possible today, except that layer squashing will not be required. | ||
|
|
||
| Future extensions to this enhancement could enable the use case of providing build input content from a persistent volume and allowing the build to store/cache content for future builds on such a volume. | ||
| Those will be discussed in the future enhancement at that time. | ||
|
|
||
| ### Implementation Details/Notes/Constraints [optional] | ||
|
|
||
| We will need to introduce a new mechanism in the build api which allows the user to indicate that they want to inject "volume" content into the build. | ||
| Initially the only allowed volume types will be ConfigMaps, Secrets, and ephemeral CSI volumes. | ||
| The api will otherwise be similar to the existing secret/configmap injection api in which users identify the configmap/secret and the target path for injection. | ||
|
|
||
| This will be done by adding our own `Volume[]` field to the Source and Docker strategy structs. | ||
| The `Volume[]` field will allow defining volumes to be mounted to the build pod in the same way that a normal pod allows for this. | ||
| However, the types of volumes that can be defined will be restricted to those explictly supported by Builds. | ||
| Similarly a `VolumeMount[]` field will be added, but without the MountPropagation and SubPath fields. | ||
| MountPropagation and SubPath can be considered for support in the future. | ||
|
|
||
| These fields will be wired, via the build controller, directly to the build pod that is constructed. | ||
| In addition all mounts into the pod will be done at a path of our choosing, not the VolumeMount path specified, to ensure the user cannot overwrite critical function inside the build pod and use it as an escalation pathway. | ||
| Example - all volumes are mounted under `/var/run/openshift.io/volumes` in the build pod containers. | ||
|
|
||
| The logic that invokes buildah will then pass the mounted directories as transient volume mount arguments. | ||
| The mount path provided to buildah will be determined from the VolumeMount specification. | ||
| Arguments such as "read only" will also be inferred. | ||
|
|
||
| Builds today have default mounts that are provided via other means: | ||
|
|
||
| 1. Entitlement data from the node, mounted at `/run/secrets/etc-pki-entitlement`, `/run/secrets/redhat.repo`, and `run/secrets/rhsm` inside the build pod. | ||
| 2. The cluster trust bundle - this is optionally mounted at `/etc/pki/ca-trust` via the BuildConfig spec's `mountTrustedCA` option. | ||
|
|
||
| If a build volume's mount path conflicts with any of the paths specified above, the build volume mount will override the default. | ||
|
|
||
| Proposed api/structs: | ||
| Note: DockerBuildStrategy will be updated in the same way. | ||
|
|
||
| ```go | ||
| // SourceBuildStrategy defines input parameters specific to an Source build. | ||
| type SourceBuildStrategy struct { | ||
| // From is reference to an DockerImage, ImageStream, ImageStreamTag, or ImageStreamImage from which | ||
| // the docker image should be pulled | ||
| From kapi.ObjectReference | ||
|
|
||
| // PullSecret is the name of a Secret that would be used for setting up | ||
| // the authentication for pulling the Docker images from the private Docker | ||
| // registries | ||
| PullSecret *kapi.LocalObjectReference | ||
|
|
||
| // Env contains additional environment variables you want to pass into a builder container. | ||
| Env []kapi.EnvVar | ||
|
|
||
| // Scripts is the location of Source scripts | ||
| Scripts string | ||
|
|
||
| // Incremental flag forces the Source build to do incremental builds if true. | ||
| Incremental *bool | ||
|
|
||
| // ForcePull describes if the builder should pull the images from registry prior to building. | ||
| ForcePull bool | ||
|
|
||
| // Volumes is a list of volumes that can be mounted by the build | ||
| // Only a subset of Kubernetes Volume sources are supported by builds. | ||
| // More info: https://kubernetes.io/docs/concepts/storage/volumes | ||
| Volumes []BuildVolume | ||
|
|
||
| // VolumeMounts are volumes to mount into the build | ||
| VolumeMounts []VolumeMount | ||
| } | ||
| ``` | ||
|
|
||
| ```go | ||
| // BuildVolume describes a volume that is made available to build pods, such that it can be mounted into the build execution environment. | ||
| // Only a subset of Kubernetes Volume sources are supported by builds. | ||
| type BuildVolume struct { | ||
|
|
||
| // Volume's name. | ||
| // Must be a DNS_LABEL and unique within the pod. | ||
| // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names | ||
| Name string `json:"name" protobuf:"bytes,1,opt,name=name"` | ||
|
|
||
| // BuildVolumeSource represents the location and type of the mounted volume. | ||
| BuildVolumeSource `json:",inline" protobuf:"bytes,2,opt,name=volumeSource"` | ||
| } | ||
|
|
||
|
|
||
| // Represents the source of a volume to mount. | ||
| // Only one of its members may be specified. | ||
| // This is a subset of the core Kubernetes VolumeSource. | ||
| type BuildVolumeSource struct { | ||
| // Secret represents a secret that should populate this volume. | ||
| // More info: https://kubernetes.io/docs/concepts/storage/volumes#secret | ||
| // +optional | ||
| Secret *kapi.SecretVolumeSource `json:"secret,omitempty" protobuf:"bytes,1,opt,name=secret"` | ||
|
|
||
| // ConfigMap represents a configMap that should populate this volume | ||
| // +optional | ||
| ConfigMap *kapi.ConfigMapVolumeSource `json:"configMap,omitempty" protobuf:"bytes,2,opt,name=configMap"` | ||
|
|
||
| // CSI (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers (Beta feature). | ||
| // +optional | ||
| CSI *kapi.CSIVolumeSource `json:"csi,omitempty" protobuf:"bytes,3,opt,name=csi"` | ||
| } | ||
| ``` | ||
|
|
||
| ```go | ||
| // VolumeMount describes a mounting of a Volume within the build execution environment. | ||
| type VolumeMount struct { | ||
| // This must match the Name of a Volume. | ||
| Name string | ||
| // Mounted read-only if true, read-write otherwise (false or unspecified). | ||
| // Defaults to false. | ||
| // +optional | ||
| ReadOnly bool | ||
| // Path within the build environment at which the volume should be mounted. Must | ||
| // not contain ':'. | ||
| MountPath string | ||
| } | ||
| ``` | ||
|
|
||
| Example usage (use of the existing secret/configmap injection api is included for comparison, it is not changing) | ||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| items: | ||
| - apiVersion: build.openshift.io/v1 | ||
| kind: BuildConfig | ||
| metadata: | ||
| name: mybuild | ||
| namespace: p1 | ||
| spec: | ||
| strategy: | ||
| sourceStrategy: | ||
| from: | ||
| kind: ImageStreamTag | ||
| name: nodejs:10-SCL | ||
| namespace: openshift | ||
| volumes: | ||
| - name: secret | ||
| secret: | ||
| secretName: somesecret | ||
| - name: config | ||
| configMap: | ||
| name: someconfigmap | ||
| items: | ||
| - key: somekey | ||
| path: volume/path/value.txt | ||
| - name: content-access | ||
| csi: | ||
| driver: csi-driver-projected-resource.openshift.io | ||
| volumeAttributes: | ||
| share: etc-pki-entitlement | ||
| volumeMounts: | ||
| - name: config | ||
| mountPath: /tmp/config | ||
| - name: secret | ||
| mountPath: /tmp/secret | ||
| - name: content-access | ||
| mountPath: /etc/pki/entitlement | ||
| type: Source | ||
| source: | ||
| secrets: | ||
| - secret: | ||
| name: myOtherSecret | ||
| destinationDir: /tmp/othersecret | ||
| configMaps: | ||
| - configMap: | ||
| name: myOtherConfigMap | ||
| destinationDir: /tmp/otherconfig | ||
| ``` | ||
| ### Risks and Mitigations | ||
| Since the build pod is privileged, we need to ensure that users cannot abuse this api to trick the build controller into creating build pods that can exploit those privileges. | ||
| This means ensuring that any volume mount specifications the user provides, which are translated into volumemounts in the build pod, cannot be used to alter the build logic. | ||
| To this end, we should explicitly control where the volumes are mounted within the build pod.(We can mount them anywhere the user specifies within the buildah container). | ||
| We also need to ensure that the user can't use this api to inject/mount content that they could not normally mount into a pod they created themselves. | ||
| For this reason we must explicitly disallow `HostPath` volume types, for example. | ||
| We will mitigate this by only supporting a subset of VolumeSources that can be mounted into builds. | ||
| As additional types are requested, we will need to determine it is safe to add them. | ||
| Gating will also ensure that we address any volume lifecycle concerns (for example - binding of PersistentVolumeClaims). | ||
|
|
||
| ## Design Details | ||
|
|
||
| ### Test Plan | ||
|
|
||
| This feature will need new e2e tests that leverage the new api option. | ||
| We have existing tests for configmap+secret injection would should be able to be copied+adapted to testing this feature relatively easily. | ||
| Tests for the CSI volume may have to be done independently of the core OpenShift test suite at outset. | ||
|
|
||
|
|
||
| ### Graduation Criteria | ||
|
|
||
| This should be introduced directly as a GA feature when it is implemented. | ||
|
|
||
| ### Upgrade / Downgrade Strategy | ||
|
|
||
| N/A | ||
|
|
||
| ### Version Skew Strategy | ||
|
|
||
| N/A | ||
|
|
||
| ## Implementation History | ||
|
|
||
| Major milestones in the life cycle of a proposal should be tracked in `Implementation | ||
| History`. | ||
|
|
||
| ## Drawbacks | ||
|
|
||
| Additional user complexity in choosing when to enable this behavior. | ||
| It is also unfortunate we can't default it because it will be a better choice for most users. | ||
|
|
||
| ## Alternatives | ||
|
|
||
| Updating the existing injection apis to have a "asVolume" field was considered as it would be a simpler implementation (more code reuse) but it was rejected as there is a long term goal to allow builds to mount traditional volumes as well. | ||
| The existing injection api can't easily be extended to support such a thing, so the design proposed in this enhancement is a better stepping stone to that goal. | ||
| This also puts us on a better path to support using volumes for caching build content between build invocations which has been a longtime goal of the build api. | ||
|
|