Allow extending reconciliation metric with custom user-defined attribute/s

[or-shachar]

We just started using Grafana alerts on condition : gotk_reconcile_condition{type="Ready", status=="False"} for > 10m.
The Grafana alert routing allows us to select the target channel based on metric attributes.

At the moment the attribute of the metrics are strictly {"kind", "name", "namespace", "type", "status"}. I wonder if we can define in the HelmRelease/Kustomization a small map of one additional value to be added to this metric. For example:

{
   "name": "billing-server",
   "namespace": "default",
   "type": "Ready",
   "status": "False",
   "team": "billing"
}

This would hint Grafana to send this alert to #billing-dev-alerts on Slack.

Without it - and assuming that we don't want to change the namespace just for the alerting - all alerts would be received in one single place and someone would have to manually dispatch them.

I saw here #1796 @seh answer about high cardinality - but let's say that the set of attributes and values would be somewhat static and users can choose not to use them at all.

WDYT?

[kingdonb]

There may be something that can be done – @pjbgf has more details

Ideally we do not need to make a change on the Flux side, but you would place some annotations or labels on the monitored objects, and the label would enrich the metrics.

[pjbgf]

I am under the impression that this can be achieved using prometheus' scrape_configs.relabel_configs:
https://github.com/prometheus/prometheus/blob/main/config/testdata/conf.good.yml#L77

Maybe using an annotation/label on the object as input.

[kingdonb]

We had a discussion about this topic at the Flux Bug Scrub today; linking the Slack thread for reference, there is some more detail in the discussion there:

https://cloud-native.slack.com/archives/CLAJ40HV3/p1658935527216089?thread_ts=1658921813.345429&cid=CLAJ40HV3

Welcome to come to the CNCF #flux channel and engage with us there. It seems like there are some folks who know the secrets required to make this enrichment configuration work, it should not require any changes to Flux.

[kingdonb]

Please refer to:

• Support for flux custom metrics #4128

This has been addressed, now you can create any metrics that you wish if the information is in the CRD spec or status, you can create a metric to report on it. There is an example in the docs specifically covering your issue now, how to get a label value and make a metric out of it.

See also:

https://fluxcd.io/flux/monitoring/custom-metrics/