Kubernetes metadata enrichment

[axw]

When running instrumented applications within a Kuberentes environment, our agents will extract a limited amount of Kubernetes pod/node metadata from cgroup info: https://github.com/elastic/apm/blob/main/specs/agents/metadata.md#containerkubernetes-metadata

It may be useful to enrich this information with additional details that can only be obtained by communicating with the Kubernetes API Server, e.g. the Kubernetes service name, and labels/annotations.

Things to research:

• can we use https://www.elastic.co/guide/en/fleet/master/kubernetes-provider.html?
• should we do the mapping in APM Server, or would it be better to store the Kubernetes topology in Elasticsearch independently, and join at query time?

[axw]

Another thing to consider would be to use a mutating admission controller to inject KUBERNETES_* environment variables. This could be a custom webhook, or it could be a published Kyverno mutate policy.