Request to make the default scraping metrics configuration configurable via helm values

[alexinfoblox]

Hello,
We are migrating from Prometheus to k8s-monitoring, and we would like to make a slight modification to the default scraping metrics configuration, which is currently explicitly defined and cannot be overridden.

For example:
Default Kube API scrapping rule in helm template

// API Server 
discovery.relabel "apiserver" {
  targets = discovery.kubernetes.services.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    regex = "default"
    action = "keep"
  }
  rule {
    source_labels = ["__meta_kubernetes_service_name"]
    regex = "kubernetes"
    action = "keep"
  }
  rule {
    source_labels = ["__meta_kubernetes_service_port_name"]
    regex = "https"
    action = "keep"
  }
....

prometheus.scrape "apiserver" {
  job_name   = "integrations/kubernetes/apiserver"
  targets    = discovery.relabel.apiserver.output
  scheme     = "https"
  scrape_interval = {{ .Values.metrics.apiserver.scrapeInterval | default .Values.metrics.scrapeInterval | quote }}
  bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  tls_config {
    insecure_skip_verify = true
  }

Allowing customization of the scraping metrics configuration via Helm values would be nice.
In this snippet we would like to make the following changes:

discovery.relabel "apiserver" {
  targets = discovery.kubernetes.endpoints.targets

rule {
    source_labels = ["__meta_kubernetes_endpoint_port_name"]
    regex = "https"
    action = "keep"
  }

Allowing changes to the default job names and define them in the values, as many of our Grafana dashboards rely on job labels with the current job names. We are not planning to change these job names in dashboard configs.
Something like :

prometheus.scrape "apiserver" {
  job_name   = {{ .Values.metrics.apiserver.jobName }}

We don't want to entirely redefine this default config via extraConfig helm value.

Could you please update config templates in the chart?

[earimont-ib]

Looking forward to this change being implemented

[petewall]

Can you help me understand the benefit of using endpoints rather than the service?
What other fields are you hoping to be able to modify? Just the job name?

[alexinfoblox]

Hello @petewall.
In this particular example of apiserver discovery, we prefer to use endpoints rather than the service to monitor individual kube-apiserver pods behind the service. This approach provides more granular and accurate metrics compared to using the service itself, which is crucial for us.

So far, modifying the job names would be sufficient

Thank you!

[petewall]

Thanks @alexinfoblox ! This is helpful.

I'll create an issue and PR to make the job labels configurable. As for changing the API server targets from services to endpoints, it does make some sense, but I'd like to coordinate with one of my colleagues on this. We're going to be focussing more on control plane monitoring this quarter.

cc @skl