Kubernetes admission controller for APM agent attachment

[axw]

We would like to implement a mutating admission webhook that automatically instruments supported runtimes.

This will initially be experimental, potentially building off the POC done by @eyalkoren in https://github.com/eyalkoren/k8s-tracing-webhook

See elastic/apm#385 for more details.

[axw]

I've started looking at how we can use Elastic Agent's Kubernetes Leadership Election provider, to avoid having multiple APM Servers (e.g. when running as a Daemonset or a Service) double up on modifying pods.

The basic build blocks are there in Elastic Agent, but not yet in Fleet. I think a more expedient approach would be to make the webhook idempotent, e.g. check for the existence of environment variables that we would add (JAVA_TOOL_OPTIONS, NODE_OPTIONS, ...) before adding them, or add our own annotation and check for that.

Moreover, https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#reinvocation-policy states:

Mutating webhooks must be idempotent, able to successfully process an object they have already admitted and potentially modified. This is true for all mutating admission webhooks, since any change they can make in an object could already exist in the user-provided object, but it is essential for webhooks that opt into reinvocation.

So we should make it idempotent anyway.

Long term we should still figure out how we can use Elastic Agent providers with Fleet, e.g. for #7161. Also, even if it's safe to have every APM Server register the same webhook, it's probably not desirable as it will add overhead to Kubernetes performance.

[axw]

See https://github.com/elastic/apm-mutating-webhook (technical preview)