Migrate // +kubebuilder:default annotations to a defaulting webhook

[skhalash]

Description

Introduce a defaulting webhook for all 3 types of pipelines and migrate away from // +kubebuilder:default annotations.

Reasons

Currently, we use // +kubebuilder:default annotations to set default values, but this approach has several limitations:

• The defaulting mechanism only triggers when the immediate parent of the field with the default annotation is explicitly set in the manifest. However, we need to apply defaults regardless of the parent’s presence. To achieve this, we end up duplicating code at each level, as seen in the following example:

// MetricPipelineRuntimeInput defines the runtime scraping section.
type MetricPipelineRuntimeInput struct {
	// If enabled, runtime metrics are scraped. The default is `false`.
	Enabled bool `json:"enabled,omitempty"`
	// Describes whether runtime metrics from specific namespaces are selected. System namespaces are disabled by default.
	// +optional
	// +kubebuilder:default={exclude: {kyma-system, kube-system, istio-system, compass-system}}
	Namespaces *MetricPipelineInputNamespaceSelector `json:"namespaces,omitempty"`
	// Describes the Kubernetes resources for which runtime metrics are scraped.
	// +optional
	// +kubebuilder:default={pod: {enabled: true}, container: {enabled: true}, node: {enabled: false}, volume: {enabled: false}}
	Resources *MetricPipelineRuntimeInputResources `json:"resources,omitempty"`
}

// MetricPipelineRuntimeInputResources describes the Kubernetes resources for which runtime metrics are scraped.
type MetricPipelineRuntimeInputResources struct {
	// Configures Pod runtime metrics scraping.
	// +optional
	// +kubebuilder:default={enabled: true}
	Pod *MetricPipelineRuntimeInputResourceEnabledByDefault `json:"pod,omitempty"`
	// Configures container runtime metrics scraping.
	// +optional
	// +kubebuilder:default={enabled: true}
	Container *MetricPipelineRuntimeInputResourceEnabledByDefault `json:"container,omitempty"`
	// Configures Node runtime metrics scraping.
	// +optional
	// +kubebuilder:default={enabled: false}
	Node *MetricPipelineRuntimeInputResourceDisabledByDefault `json:"node,omitempty"`
	// Configures Volume runtime metrics scraping.
	// +optional
	// +kubebuilder:default={enabled: false}
	Volume *MetricPipelineRuntimeInputResourceDisabledByDefault `json:"volume,omitempty"`
}

// MetricPipelineRuntimeInputResourceEnabledByDefault defines if the scraping of runtime metrics is enabled for a specific resource. The scraping is enabled by default.
type MetricPipelineRuntimeInputResourceEnabledByDefault struct {
	// If enabled, the runtime metrics for the resource are scraped. The default is `true`.
	// +optional
	// +kubebuilder:default=true
	Enabled *bool `json:"enabled,omitempty"`
}

• This setup requires full end-to-end tests, which are both slow and costly.
• Common values like the system namespace list must be repeated across multiple places, leading to unnecessary code duplication.
• We introduce unnecessary types like MetricPipelineRuntimeInputResourceEnabledByDefault and MetricPipelineRuntimeInputResourceDisabledByDefault, just to manage different sets of annotations.

An alternative approach is to use a defaulting webhook. While maintaining webhook infrastructure (e.g., certificates) can be costly, it's already in place for converting and validating webhooks, and is likely to stay in the future.

First Step:
Kustomize for Webhook Configurations: We can consider moving the ValidatingWebhookConfiguration from the operator code to Kustomize. The operator would then only patch the CA bundle, while the rest of the static configuration would be managed via Kustomize. The lifecycle would be controlled by a lifecycle manager. The same approach can be applied to the MutatingWebhookConfiguration for defaulting.

• All webhook resources are part of kustomize (without caBundle section)
• The manager only patches the resources with caBundle
• There are no leftover resources after an upgrade to the new version
• Validation should work as before, mutating logic gets triggered with NOOP

Second Step
Migrate all defaulting logic from CRDs to mutating webhooks

• No kubebuilder annotations for defaulting present anymore
• Defaulting log works as before