feat: disable labels and annotations metrics when metric-annotations-…

[opeco17]

What this PR does / why we need it:

kube-state-metrics provides a lot of kube_*_annotations and kube_*_labels by default, and it consumes memory significantly.

kube_pod_annotations{instance="...:8080", job="monitoring/kube-state-metrics", namespace="namespace", pod="podname-bc94dfc78-fcf4j", uid="a482250c-6f6e-4533-a596-3eff484d8969"}

By disabling those metrics by default, kube-state-metrics can reduce memory usage.

Those metrics are provided when kube_*_annotations and kube_*_labels are configured.

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)

decrease

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1829

[k8s-ci-robot]

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dgrisonnet]

That's a really cool change @opeco17, thank you!

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgrisonnet, opeco17

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dgrisonnet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mrueg]

@dgrisonnet @opeco17 Is this a breaking change to the previous behaviour?

[dgrisonnet]

It is indeed a breaking change. I was a bit too quick with merging this one.

Although this is a breaking change, I wonder if users really rely on these empty metrics since the information inside of them are already present in other metrics such as the more commonly used _info.

That said, I looked at the options' doc and was expecting that we are saying that we are not exposing labels/annotations metrics when the allowlist are empty. But in the end we say:

By default the metric contains only name and namespace labels.

[dgrisonnet]

These metrics being EXPERIMENTAL, I would personally move ahead with this change and update the doc accordingly.

What is your opinion on that @mrueg @rexagod?

[mrueg]

This works for me, we should mention it in the changelogs though as a breaking change.

[rexagod]

#2145 (comment) makes sense. Even if this is a breaking change, we aren't doing anything that goes against the laid out commitments. +1.

[cykl]

@dgrisonnet

Although this is a breaking change, I wonder if users really rely on these empty metrics

I will put an end to the suspense. At least one user really relied on these metrics, me. 😁

since the information inside of them are already present in other metrics such as the more commonly used _info.

That doesn't sound right to me. AFAIK The only way to retrieve labels associated with a kubernetes object is the kube_*_labels metrics. In my case, some labels are required to aggregate some metrics or route some alerts.

It seems a very common need, and I expect that others have been impacted too (for example search for "kubernetes prometheus cronjob" and you will find dozen of examples relying on them).

While release notes mention it, I'm a bit surprised that kube-state-metrics introduced this breaking changes with such lightness.