diff --git a/keps/prod-readiness/sig-api-machinery/2887.yaml b/keps/prod-readiness/sig-api-machinery/2887.yaml new file mode 100644 index 00000000000..7955aae9bad --- /dev/null +++ b/keps/prod-readiness/sig-api-machinery/2887.yaml @@ -0,0 +1,3 @@ +kep-number: 2887 +alpha: + approver: "@deads2k" diff --git a/keps/sig-api-machinery/2887-openapi-enum-types/README.md b/keps/sig-api-machinery/2887-openapi-enum-types/README.md new file mode 100644 index 00000000000..e9c4a3b0501 --- /dev/null +++ b/keps/sig-api-machinery/2887-openapi-enum-types/README.md @@ -0,0 +1,838 @@ + +# KEP-2887: OpenAPI Enum Types + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [Enum Pattern](#enum-pattern) + - [Enum Tag](#enum-tag) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [OpenAPI v2 Breakage](#openapi-v2-breakage) + - [Enum Fields Pruning for Feature Disablement](#enum-fields-pruning-for-feature-disablement) + - [Test Plan](#test-plan) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Beta](#beta) + - [Stable](#stable) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + - [Do Not Break Existing Clients](#do-not-break-existing-clients) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [X] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [X] (R) KEP approvers have approved the KEP status as `implementable` +- [X] (R) Design details are appropriately documented +- [X] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests for meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [X] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [X] (R) Production readiness review completed +- [X] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +This KEP defines Enum Types support for OpenAPI. Currently, types in our API have fields +that are actually enums but represented as plain string. This KEP proposes a `+enum` +marker to type aliases that represent enums. The OpenAPI generator will have +the capability to recognize the marker and auto-detect possible values for an enum. + +## Motivation + + + +Currently, types that are actually enums are represented as plain strings. For example, +here is the definition of `Protocol`, which must be one of `TCP`, `UDP`, and `SCTP`. +```go +package core +// ... + +// Protocol defines network protocols supported for things like container ports. +type Protocol string + +const ( + // ProtocolTCP is the TCP protocol. + ProtocolTCP Protocol = "TCP" + // ProtocolUDP is the UDP protocol. + ProtocolUDP Protocol = "UDP" + // ProtocolSCTP is the SCTP protocol. + ProtocolSCTP Protocol = "SCTP" +) + +// ContainerPort represents a network port in a single container +type ContainerPort struct { + // ... + + // Required: Supports "TCP", "UDP" and "SCTP" + // +optional + Protocol Protocol + + // ... +} +``` + +In order to convey that the `Protocol` type is an enum, every field of the type +have to comment its values like `ContainerPort` does. The same goes for validation - +protocol being one of "TCP", "UDP", and "SCTP" is duplicated among the code base. + +Tools and UIs can also benefit from the additional piece of information. For example, +Protocol can be shown as a dropdown of possible values instead of a textbox. +The user no longer needs to see an error of invalid value after submits the request. + +### Goals + + + +- Add a marker that indicates a field as an enum. +- Make the Kubernetes OpenAPI generator recognize the marker. +- Deduce values of an enum type from its Go type definition. +- Detect and annotate enum types in all built-in types. +- Unify the marker with that of kube-builder's enum mechanism. + +### Non-Goals + + + +This KEP does not provide support for validation that is more complicated than an enum type +like regex or prefix. Instead, use other validation mechanism that is or will be supported. + +This KEP treat any enum type as a closed enum. Any change to the set of possible values should be +treated as creating a new type and thus should follow the standard procedure of changing an API. + +## Proposal + +When generating OpenAPI schema, the de facto enum types should be reflected as enums in the resulting schema. + +We define enum types with following properties: +- All enum types are closed, i.e., the type includes a finite number of possible values. + However, for compatibility, a client may allow unknown values from responses. +- All enum types should be string. There can be no enum integers, floats, etc + +Additionally, This KEP makes the following assumptions on the implementation: +- The canonical definition of a type is written in go and follows the standard + Kubernetes convention. +- All enum types have their own type alias. i.e. , no raw string enums. + +### Enum Pattern + +The Enum Pattern of a type, appearing in a go source file, is defined as follows: +```go +package internal + +// FooType defines a type alias, underlying type must be string +type FooType string + +// in the same package, define constants with the alias +const ( + BarConst FooType = "Bar" + BazConst FooType = "Baz" +) + +// in one or more resources or nested objects, one of the fields is of +// the defined alias type. + +// ExampleResource should refer FooType +type ExampleResource struct { + FooField FooType +} + +``` +Enum Type is the type that follows such pattern (`FooType` in the example above). +Enum Values are the possible values of the Enum Type (`Bar` and `Baz` are Enum Values for `FooType`). + +### Enum Tag +The Enum Tag is the marker `// +enum` that must be in the closest comment blocks +of an enum type following Enum Pattern. + +```go +package internal + +// FooType define sa type alias, underlying type must be string +// +enum +type FooType string +``` + +An actual example looks like +```go +package core + +// Protocol defines network protocols supported for things like container ports. +// +enum +type Protocol string +``` + + + +### Notes/Constraints/Caveats (Optional) + + + +Not all types following the Enum Pattern are actually enum types. Here is a counter-example. +```go +package core + +// StorageMedium defines ways that storage can be allocated to a volume. +type StorageMedium string + +// These are the valid value for StorageMedium +const ( + StorageMediumDefault StorageMedium = "" // use whatever the default is for the node + StorageMediumMemory StorageMedium = "Memory" // use memory (tmpfs) + StorageMediumHugePages StorageMedium = "HugePages" // use hugepages + StorageMediumHugePagesPrefix StorageMedium = "HugePages-" // prefix for full medium notation HugePages- +) +``` + +Here, `StorageMedium` can have infinite number of possible values, which disqualify it as an enum type. + +### Risks and Mitigations + + + +The syntax proposed in this KEP is different from existing implementation of kube-builder. +Namely, kube-builder requires possible values in the marker itself, and is treated +as a validation. +```go +package example + +// Alias +// +kubebuilder:validation:Enum=Lion;Wolf;Dragon +type Alias string +``` +The two approaches are compatible, but eventually they will have to be unified. +To retain compatibility with existing users of kube-builder, the auto-deduction will be added +to kube-builder once the work is done with built-in types. + +## Design Details + + + +### OpenAPI v2 Breakage +This feature, when enabled, breaks existing code using swagger to generate client or server stub from the spec +for languages with built-in enum support, including Java and C#. +Take the following spec as an example. + +```yaml +definitions: + Foo: + type: object + properties: + bar: + type: string + enum: + - A + - B + - C +``` + +The generated Java object looks like +```java +public class Foo { + + @JsonAdapter(BarEnum.Adapter.class) + public enum BarEnum { + A("A"), + B("B"), + C("C"); + // ... + } + private BarEnum bar = null; +} +``` +If the `enum` field was not in the definition, the `Bar` field would be a String. +Maintainers of the clients should include the change in a new major release. + +If, rather than update code generator or use of the generated client, a client would prefer enum fields to be strings as they were previously, a simple preprocessor +will be provided to prune the enum information similar to that of `api-server` (see below). + +### Enum Fields Pruning for Feature Disablement + +If the feature is disabled through the feature gate, `api-server` will remove all aspects about +enum types from the schema, making the result identical to that of the prior format. + +After this feature graduates to GA, the pruning code will be removed from `api-server`. + +### Test Plan + + + +Testing should focus on `kubernetes/kube-openapi` repository. There will be +integration tests against cases with valid and invalid definition of enum types. + +In the `kubernetes/kubernetes` repository, as part of OpenAPI v3 test suites, +there should be integration and e2e tests that validate present of enum fields. + +### Graduation Criteria + +#### Alpha + + - Feature implemented behind the feature gate + - Initial tests, both in k/k and k/kube-openapi, are completed and enabled + +#### Beta + + - All enums in Kubernetes built-in types have enum tags properly added. + - Enum syntax of kube-builder updated to match that of this KEP. + +#### Stable + + - OpenAPI-level validation of enum types in built-in types removed in favor of that provided by the schema. + +### Upgrade / Downgrade Strategy +Enable/disable the OpenAPIEnum feature gate. + +### Version Skew Strategy +The API is still compatible even with enum types enabled. +For the conversion from v2 to v3 (see OpenAPI v3 KEP), enum types will be carried over and thus +should not change the behavior of v3. + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [X] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: `OpenAPIEnum` + - Components depending on the feature gate: `kube-apiserver` +- [ ] Other + - Describe the mechanism: + - Will enabling / disabling the feature require downtime of the control + plane? + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? (Do not assume `Dynamic Kubelet Config` feature is enabled). + +###### Does enabling the feature change any default behavior? + + + +Yes. Any clients generated from the new schema will see enum fields. +For languages with enum support, OpenAPI language-level generator (swagger) will create +Language-specific enum types instead of plain strings. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +Yes. By disabling the feature gate and restart `api-server`. +###### What happens if we reenable the feature if it was previously rolled back? + +This feature needs access only to information immutable at run-time. There should +be no problem re-enabling it. +###### Are there any tests for feature enablement/disablement? + + +N/A +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +###### What specific metrics should inform a rollback? + + + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +No. This feature only adds enum fields to enum types. +###### Will enabling / using this feature result in introducing new API types? + + + +No. +###### Will enabling / using this feature result in any new calls to the cloud provider? + + +No. +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + +No. Enum information only appears in `/openapi/` endpoints but not standard APIs. +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + +No. +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + +No. The addition of enum fields take negligible amount of memory. +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +## Drawbacks + + +This KEP will break any users that expect enum types to be plain strings. +This is the main reason for not implementing this feature in OpenAPI v2 (see Alternative below). +## Alternatives + + + +### Do Not Break Existing Clients + +This change can go along with OpenAPI v3 instead of v2, which prevent the breakage mentioned above. +However, an affected client can release a new major version to introduce the breaking change itself +instead of waiting for new major version of OpenAPI. In this way, we can "make things right" and benefit +from the improvement much earlier. \ No newline at end of file diff --git a/keps/sig-api-machinery/2887-openapi-enum-types/kep.yaml b/keps/sig-api-machinery/2887-openapi-enum-types/kep.yaml new file mode 100644 index 00000000000..3c42ad8aead --- /dev/null +++ b/keps/sig-api-machinery/2887-openapi-enum-types/kep.yaml @@ -0,0 +1,45 @@ +title: KEP Template +kep-number: 2887 +authors: + - "@jiahuif" +owning-sig: sig-api-machinery +participating-sigs: [] +status: implementable +creation-date: 2021-08-20 +reviewers: + - "@apelisse" + - "@jpbetz" + - "@sttts" +approvers: + - "@jpbetz" + - "@deads2k" + - "@lavalamp" +prr-approvers: + - "@deads2k" + +see-also: + - https://github.com/kubernetes/enhancements/issues/2896 +replaces: [] + +# The target maturity stage in the current dev cycle for this KEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.23" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.23" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: OpenAPIEnum + components: + - kube-apiserver +disable-supported: true + +# The following PRR answers are required at beta release +metrics: [] #TBD