From 40b07029d658a1dcfc2c2bf1f3e4c5bffe777370 Mon Sep 17 00:00:00 2001 From: Alexander Zielenski <351783+alexzielenski@users.noreply.github.com> Date: Tue, 16 Aug 2022 17:56:38 -0700 Subject: [PATCH] add blog post: immutability patterns using CEL --- .../2022-08-15-immutability-with-cel.md | 525 ++++++++++++++++++ 1 file changed, 525 insertions(+) create mode 100644 content/en/blog/_posts/2022-08-15-immutability-with-cel.md diff --git a/content/en/blog/_posts/2022-08-15-immutability-with-cel.md b/content/en/blog/_posts/2022-08-15-immutability-with-cel.md new file mode 100644 index 0000000000000..e8100f77a9585 --- /dev/null +++ b/content/en/blog/_posts/2022-08-15-immutability-with-cel.md @@ -0,0 +1,525 @@ +--- +layout: blog +title: "Enforce CRD Immutability with CEL Transition Rules" +date: 2022-08-15 +slug: enforce-immutability-using-cel +canonicalUrl: https://www.kubernetes.dev/blog/2022/08/15/enforce-immutability-using-cel +--- + +**Author:** [Alexander Zielenski (Google)](https://github.com/alexzielenski) +--- + +# Introduction +Immutable fields can be found in a few places in the built-in Kubernetes types such as PodSpec, EphemeralContainers, Finalizers, and more. + +Until recently the best way to create immutable fields for Custom Resource Definitions has been to create an admission webhook: an unfortunate tradeoff for the common case of immutable fields. + +Beta since Kubernetes 1.25, CEL Validation Rules allow CRD authors to express validation constraints on their fields using a rich expression language, [CEL](https://github.com/google/cel-spec). This article explores how you can use validation rules to implement a few common immutability patterns. +# Validation Rules Basics +The new support for CEL validation rules in Kubernetes allows CRD authors to add complicated admission logic for their resources without writing any code! + +For example, A CEL rule to constrain a field `maximumSize` to be greater than a `minimumSize` for a CRD might look like the following: +```yaml + rule: self.maximumSize > self.minimumSize +message: “Maximum size must be greater than minimum size.” +``` +The rule field contains an expression written in CEL. `self` is a special keyword in CEL which refers to the object whose type contains the rule. + +The message field is an error message which will be sent to Kubernetes clients whenever this particular rule is not satisfied. +## More Information +For more details about the capabilities and limitations of Validation Rules using CEL, please refer to [the documentation](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules). The [cel-spec](https://github.com/google/cel-spec) is also a good reference for information specifically related to the language. + +# Immutability Patterns with CEL Validation Rules +This section implements several common use cases for immutability in Kubernetes Custom Resource Definitions using validation rules expressed as [kubebuilder marker comments](https://book.kubebuilder.io/reference/markers/crd.html). + +We will also show the resultant OpenAPI generated by the kubebuilder marker comments so that those not using kubebuilder can also follow along. + +## Project Setup + +To demonstrate how to use CEL rules with kubebuilder comments, we first need to set up a Golang project structure with the CRD defined in Go. Afterwards, we will be able to use this go project to demonstrate different design patterns. + +You may skip this step if you are only interested in the resultant OpenAPI extensions. + +We begin with a folder structure of a Go module set up like the following. If you have your own project already set up feel free to adapt this tutorial to your liking: +```console +cel-immutability-tutorial +├── generate.go +├── pkg +│ └── apis +│ └── stable.example.com +│ └── v1 +│ ├── doc.go +│ └── types.go +└── tools.go +``` + +This is the typical folder structure used by Kubernetes projects for defining new API resources. + +`doc.go` contains package-level metadata such as the group and the version: +```go +// +groupName=stable.example.com +// +versionName=v1 +package v1 +``` + +`types.go` contains all type definitions in stable.example.com/v1 + +```go +package v1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// An empty CRD as an example of defining a type using controller tools +// +kubebuilder:storageversion +// +kubebuilder:subresource:status +type TestCRD struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec TestCRDSpec `json:"spec,omitempty"` + Status TestCRDStatus `json:"status,omitempty"` +} + +type TestCRDStatus struct {} +type TestCRDSpec struct { + // We will fill this in as we go along +} +``` + +`tools.go` contains a dependency on [controller-gen](https://book.kubebuilder.io/reference/generating-crd.html#generating-crds) which will be used to generate the CRD definition: + +```go +//go:build tools + +package celimmutabilitytutorial + +// Force direct dependency on code-generator so that we may use it with go run +import ( + _ "sigs.k8s.io/controller-tools/cmd/controller-gen" +) +``` + +Finally, `generate.go`contains a `go:generate` directive to make use of `controller-gen`. `controller-gen` parses our `types.go` and creates generates CRD yaml files into a `crd` folder: + +```go +package celimmutabilitytutorial + +//go:generate go run sigs.k8s.io/controller-tools/cmd/controller-gen crd paths=./pkg/apis/... output:dir=./crds +``` + + +You may now want to add dependencies for our definitions and test the code generation: + +```console +$ cd cel-immutability-tutorial +$ go mod init / +$ go mod tidy +$ go generate ./… +``` + +After running these commands we now have completed our basic project structure. The folder tree should look like this: + +``` +cel-immutability-tutorial +├── crds +│ └── stable.example.com_testcrds.yaml +├── generate.go +├── go.mod +├── go.sum +├── pkg +│ └── apis +│ └── stable.example.com +│ └── v1 +│ ├── doc.go +│ └── types.go +└── tools.go +``` + +Our test CRD yaml is now available in `crds/stable.example.com_testcrds.yaml` + +## Immutable After First Write +A common immutability design pattern is to make the field immutable once it has been first set. This example will throw a validation error if the field after changes after being first initialized. + +```go +// +kubebuilder:validation:XValidation:rule="!has(oldSelf.value) || has(self.value)", message="Value is required once set" +type ImmutableSinceFirstWrite struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // +kubebuilder:validation:Optional + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable" + // +kubebuilder:validation:MaxLength=512 + Value string `json:"value"` +} +``` + +The `+kubebuilder` directives in the comments inform controller-gen how to annotate the generated OpenAPI. The `XValidation` rule causes the rule to appear among the `x-kubernetes-validations` OpenAPI extension. Kubernetes then respects the OpenAPI spec to enforce our constraints. + +To be immutable after first write, we require the following constraints: +1. Field must be allowed to be initially unset `+kubebuilder:validation:Optional` +2. Once set, field must not be allowed to be removed: `!has(oldSelf.value) | has(self.value)` (type-scoped rule) +3. Once set, field must not be allowed to change value `self == oldSelf` (field-scoped rule) + +Also note the additional directive `+kubebuilder:validation:MaxLength`. CEL requires that all strings have attached max length so that it may estimate the computation cost of the rule. Rules that are too expensive will be rejected. For more information on CEL cost budgeting, check out the other tutorial. + +### Example Usage + +```console +# Ensure the CRD yaml is generated by controller-gen +$ go generate ./... + +$ kubectl apply -f crds/stable.example.com_immutablesincefirstwrites.yaml +customresourcedefinition.apiextensions.k8s.io/immutablesincefirstwrites.stable.example.com created + +$ kubectl apply -f - <: Invalid value: "object": Value is required once set +``` + +### Generated openAPIV3Schema +Note that in the generated schema there are two separate rule locations. One is directly attached to the property `immutable_since_first_write`. The other rule is associated with the crd type itself. +```yaml +openAPIV3Schema: + properties: + value: + maxLength: 512 + type: string + x-kubernetes-validations: + - message: Value is immutable + rule: self == oldSelf + type: object + x-kubernetes-validations: + - message: Value is required once set + rule: '!has(oldSelf.value) || has(self.value)' +``` + +## Immutable Upon Creation-Time + +A field which is immutable upon creation time is implemented similarly to the earlier example. The difference is that that +field is marked required, and the type-scoped rule is no longer necessary. + +```go +type ImmutableSinceCreation struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // +kubebuilder:validation:Required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="Value is immutable" + // +kubebuilder:validation:MaxLength=512 + Value string `json:"value"` +} +``` + +This field will be required when the object is created, and after that point will not be allowed to be modified. Our CEL Validation Rule `self == oldSelf` + +### Usage Example + +```console +# Ensure the CRD yaml is generated by controller-gen +$ go generate ./... + +$ kubectl apply -f crds/stable.example.com_immutablesincecreations.yaml +customresourcedefinition.apiextensions.k8s.io/immutablesincecreations.stable.example.com created + +$ kubectl apply -f - <: Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation + +$ kubectl apply -f - <: Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation +``` + +### Generated openAPIV3Schema +```yaml +openAPIV3Schema: + properties: + value: + maxLength: 512 + type: string + x-kubernetes-validations: + - message: Value is immutable + rule: self == oldSelf + required: + - value + type: object +``` + +## Append-Only List of Containers +In the case of ephemeral containers on Pods, Kubernetes enforces that the elements in the list are immutable, and can’t be removed. The following shows how you could use CEL to achieve the same behavior. + +```go +// +kubebuilder:validation:XValidation:rule="!has(oldSelf.value) || has(self.value)", message="Value is required once set" +type AppendOnlyList struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=100 + // +kubebuilder:validation:XValidation:rule="oldSelf.all(x, x in self)",message="Values may only be added" + Values []v1.EphemeralContainer `json:"value"` +} +``` + +1. Once set, field must not be deleted: `!has(oldSelf.value) || has(self.value)` (type-scoped) +2. Once a value is added it is not removed: `oldSelf.all(x, x in self)` (field-scoped) +2. Value may be initially unset: `+kubebuilder:validation:Optional` + +Note that for cost-budgeting purposes, `MaxItems` is also required to be specified. + +### Example Usage + +```console +# Ensure the CRD yaml is generated by controller-gen +$ go generate ./... + +$ kubectl apply -f crds/stable.example.com_appendonlylists.yaml +customresourcedefinition.apiextensions.k8s.io/appendonlylists.stable.example.com created + +$ kubectl apply -f - <: Invalid value: "object": Value is required once set +``` + +### Generated openAPIV3Schema +```yaml +openAPIV3Schema: + properties: + value: + items: ... + maxItems: 100 + type: array + x-kubernetes-validations: + - message: Values may only be added + rule: oldSelf.all(x, x in self) + type: object + x-kubernetes-validations: + - message: Value is required once set + rule: '!has(oldSelf.value) || has(self.value)' +``` + +## Map with append-only keys, immutable values + +```go +// A map which does not allow keys to be removed or their values changed once set. New keys may be added, however. +// +kubebuilder:validation:XValidation:rule="!has(oldSelf.values) || has(self.values)", message="Value is required once set" +type MapAppendOnlyKeys struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxProperties=10 + // +kubebuilder:validation:XValidation:rule="oldSelf.all(key, key in self && self[key] == oldSelf[key])",message="Keys may not be removed and their values must stay the same" + Values map[string]string `json:"values,omitempty"` +} +``` + +1. Once set, field must not be deleted: `!has(oldSelf.values) || has(self.values)` (type-scoped) +2. Once a key is added it is not removed nor is its value modified: `oldSelf.all(key, key in self && self[key] == oldSelf[key])` (field-scoped) +3. Value may be initially unset: `+kubebuilder:validation:Optional` + +### Example Usage + +```console +# Ensure the CRD yaml is generated by controller-gen +$ go generate ./... + +$ kubectl apply -f crds/stable.example.com_mapappendonlykeys.yaml +customresourcedefinition.apiextensions.k8s.io/mapappendonlykeys.stable.example.com created + +$ kubectl apply -f - <: Invalid value: "object": Value is required once set +``` + +### Generated openAPIV3 Schema + +```yaml +openAPIV3Schema: + description: A map which does not allow keys to be removed or their values + changed once set. New keys may be added, however. + properties: + values: + additionalProperties: + type: string + maxProperties: 10 + type: object + x-kubernetes-validations: + - message: Keys may not be removed and their values must stay the same + rule: oldSelf.all(key, key in self && self[key] == oldSelf[key]) + type: object + x-kubernetes-validations: + - message: Value is required once set + rule: '!has(oldSelf.values) || has(self.values)' +``` + +# Going Further +The above examples showed how CEL rules can be added to kubebuilder types. The same rules can be added directly to OpenAPI if writing CRD yaml by hand. + +For native types, the same behavior can be achieved using kube-openapi’s marker [`+validations`](https://github.com/kubernetes/kube-openapi/blob/923526ac052c59656d41710b45bbcb03748aa9d6/pkg/generators/extension.go#L69). + +We have only scratched the surface of the expressiveness of validation rules using CEL. For more information please check out [the documentation](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules).