Grafana Agent kafka_exporter failing with timeout

[lekkalraja]

Hi,

We have installed Grafana Agent and enabled kafka_exporter to scrape metrics from one of our broker which is having high load.
We are getting timeout due to calculation of consumer lag and offsets, even after increasing the scrape_timeout to more than 5 mins .

But when we disabled (using group_filter) we able to fetch the metrics around 200,000 metrics for every scrape.

Please suggest reg the timeout when it's calculating lag & offset which we are suspecting that for every scrape it may pull 500,000 metrics.

Grafana Agent Config For your Reference :
server:
http_listen_address: 127.0.0.1
http_listen_port: 9090
prometheus:
wal_directory: /var/lib/agent
global:
scrape_interval: 10m
scrape_timeout: 9m
remote_write:
- url: <CORTEX_URL>
basic_auth:
username: <CORTEX_TENANT>
password: <CORTEX_PWD>
integrations:
agent:
enabled: true
node_exporter:
enabled: true
include_exporter_metrics: true
disable_collectors:
- mdadm
kafka_exporter:
enabled: true
kafka_uris: [localhost:9094]

Error:
level=warn msg="GET /integrations/kafka_exporter/metrics 35.859912977s, error: write tcp 127.0.0.1:9090->127.0.0.1:34176: i/o timeout ws: false; Accept: application/openmetrics-text; version=0.0.1,text/plain;version=0.0.4;q=0.5,/;q=0.1; Accept-Encoding: gzip; User-Agent: Prometheus/v0.22.0; X-Prometheus-Scrape-Timeout-Seconds: 540; "