Support for flux custom metrics

[darkowlzz]

We have received a lot of feedback related to flux metrics from many users over time. In a lot of the feedback users showed the need to gather more information about particular flux resources (refer #3106, #3769, #2674, #1796), for example the Git branch or tag that their resources use, or the helm chart version that's currently deployed, so that they could set alerts when these configurations change. Some users expressed the need to be able to add custom fields in the metrics (refer #2632) so that they can use it to categorize the resources by their teams or cluster. A few also wanted to expose the reasons for the objects to be in a non-ready state for an extended period of time so that they could be alerted about them. Also, some wanted high quality metrics about flux objects that are more useful for them (refer #1479).
Most of these requests couldn't be implemented within flux as they would increase the cardinality of the flux metrics and also introduce fragmentation in the common metrics shared among the different flux resources kinds. The flux metrics also exported a lot of the metrics from controller-runtime which can help monitor the state of the controllers themselves.

Most of the data needed for the above metrics requirements are present on the flux custom resources in the spec and status fields. They don't need the flux controllers to be exported. Flux controllers export metrics about the operation of the reconcilers that manage the flux resources. These metrics are used in the flux control-plane dashboard to show the state of the flux components. The custom metrics that most users ask for can be exported by an external system that can query the data from kube-apiserver, independent of the flux controllers. Kube-state-metrics (KMS) is one such tool that can be used for this. The custom-resource state metrics docs shows some examples of how to expose such metrics.

In flux v2.1, in order to allow the users to set up and configure their own metrics based on their needs, the monitoring setup and configuration provided by flux will no longer be recommended, and new guides and examples will be provided to show how kube-state-metrics can be used to achieve the same. A new version of the flux cluster dashboard will be provided as an example dashboard which will use the metrics from kube-state-metrics to create the same panels.

In addition, the flux readiness condition metrics (gotk_reconcile_condition) exported by the controllers will be modified to only export metrics for live resources. Previously, this metric would show a Deleted status for Ready when objects get deleted. Flux v2.1 will only report metric for objects that exist. The deleted object metrics will no longer be exported. This is done to reduce the stale metrics of the deleted objects and gradually stop exporting such metrics as a whole in a future release. This may affect the old cluster dashboard. Use of kube-state-metrics for similar metrics and dashboard is encouraged. The new custom metrics guide will contain examples for it.
A similar change for the HelmRepository cache event metrics (gotk_cache_events_total) will be implemented to delete the stale metrics for the deleted HelmRepositories.

To-do:

• Provide examples of new monitoring setup using kube-state-metrics in fluxcd/flux2-monitoring-example repository - Add kube-prometheus-stack, loki-stack & monitoring-config flux2-monitoring-example#1
• Add custom metrics guide website#1610
• runtime/metrics: Delete metrics on object delete pkg#612
• Delete stale metrics on object delete source-controller#1183
• Delete stale metrics on object delete kustomize-controller#944
• Delete stale metrics on object delete notification-controller#599
• Delete stale metrics on object delete image-reflector-controller#430
• Delete stale metrics on object delete image-automation-controller#570
• Delete stale metrics on object delete helm-controller#753
• Add monitoring configuration deprecation notice #4169