inputs.prometheus missing histogram infinity bucket when protobuf protocol is used

[mmolnar]

Relevant telegraf.conf

[[outputs.file]]
  files = ["collector.out"]

[[inputs.prometheus]]
  urls = ["http://127.0.0.1:9305/metrics"]
  metric_version = 2

Logs from Telegraf

2022-07-12T09:09:53Z I! Starting Telegraf 1.24.0-50848315
2022-07-12T09:09:53Z I! Loaded inputs: prometheus
2022-07-12T09:09:53Z I! Loaded aggregators: 
2022-07-12T09:09:53Z I! Loaded processors: 
2022-07-12T09:09:53Z I! Loaded outputs: file
2022-07-12T09:09:53Z I! Tags enabled: host=molnar
2022-07-12T09:09:53Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"molnar", Flush Interval:10s
2022-07-12T09:09:53Z D! [agent] Initializing plugins
2022-07-12T09:09:53Z D! [agent] Connecting outputs
2022-07-12T09:09:53Z D! [agent] Attempting connection to [outputs.file]
2022-07-12T09:09:53Z D! [agent] Successfully connected to outputs.file
2022-07-12T09:09:53Z D! [agent] Starting service inputs
2022-07-12T09:10:00Z D! [outputs.file] Wrote batch of 1000 metrics in 7.97117ms
2022-07-12T09:10:00Z D! [outputs.file] Buffer fullness: 157 / 10000 metrics

System info

Telegraf 1.24, Ubuntu 18.04

Docker

No response

Steps to reproduce

1. Have an service capable of responding in protobuf protool, such as prometheus grok exporter
2. Collect metrics using inputs.prometheus
3. Output metrics using output.prometheus_client

Expected behavior

Collected histogram metrics will work as proper histogram when ingested by Prometheus.

Actual behavior

Collected histogram metric do not have le=+Inf bucket which is required for Prometheus to properly use histogram and histogram_quantile function

Additional info

Exposed metrics in text exposition format:

# HELP middleware_requests_latency_seconds Histogram of HTTP requests latency.
# TYPE middleware_requests_latency_seconds histogram
middleware_requests_latency_seconds_bucket{status="200",le="0.01"} 0
middleware_requests_latency_seconds_bucket{status="200",le="0.05"} 10
middleware_requests_latency_seconds_bucket{status="200",le="0.1"} 11
middleware_requests_latency_seconds_bucket{status="200",le="0.5"} 11
middleware_requests_latency_seconds_bucket{status="200",le="1"} 11
middleware_requests_latency_seconds_bucket{status="200",le="5"} 11
middleware_requests_latency_seconds_bucket{status="200",le="10"} 11
middleware_requests_latency_seconds_bucket{status="200",le="50"} 11
middleware_requests_latency_seconds_bucket{status="200",le="+Inf"} 11
middleware_requests_latency_seconds_sum{status="200"} 0.265
middleware_requests_latency_seconds_count{status="200"} 11

Collected metrics in influxdb format:

prometheus,le=0.01,status=200 middleware_requests_latency_seconds_bucket=0 
prometheus,le=0.05,status=200 middleware_requests_latency_seconds_bucket=10 
prometheus,le=0.1,status=200 middleware_requests_latency_seconds_bucket=11 
prometheus,le=0.5,status=200 middleware_requests_latency_seconds_bucket=11 
prometheus,le=1,status=200 middleware_requests_latency_seconds_bucket=11 
prometheus,le=5,status=200 middleware_requests_latency_seconds_bucket=11 
prometheus,le=10,status=200 middleware_requests_latency_seconds_bucket=11 
prometheus,le=50,status=200 middleware_requests_latency_seconds_bucket=11 
prometheus,status=200 middleware_requests_latency_seconds_count=11,middleware_requests_latency_seconds_sum=0.265

According to the protobuf protocol definition for prometheus metrics the Infinity bucket can be optional, this is most likely because its value should be allways the same as histogram sample count, so it can be calculated.

The text exposition format does not make the bucket optional, so simply parsing, passing along and then writing to text format of the same metrics does not work and breaks histograms.

Telegraf should ensure that the infinity bucket is allways present on input, creating it from sample count if necesary.

[powersj]

next steps: (finish) review of #11486