[prometheus-blackbox-exporter] not probing the targets automatically

[meektechie]

Describe the bug a clear and concise description of what the bug is.

We have deployed blackbox-expoerter through helm chart in k8s (EKS), the helm deployment is via argoCD. Once deployed it is not probing external targets mentioned on serviceMonitor: targets:.

Without argoCD, while deploying manually it works like a charm.

I have tried most of the ways, like manually probing the targets (it works fine), and upgrading and downgrading the versions. but no luck.

Kindly help me

What's your helm version?

v3.12.2

What's your kubectl version?

v1.28.2

Which chart?

prometheus-blackbox-exporter

What's the chart version?

8.4.0

What happened?

blackbox-expoerter not probing the targets automatically

What you expected to happen?

it should probe the targets on service-monitor and expose the metrics

How to reproduce it?

Deploy blackbox-exporter helm with argoCD

Enter the changed values of values.yaml?

extraArgs:
- --log.level=debug
config:
modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
follow_redirects: true
preferred_ip_protocol: "ip4"
http_2xx4xx:
prober: http
timeout: 10s
http:
valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
method: GET
tls_config:
insecure_skip_verify: true
follow_redirects: true
preferred_ip_protocol: "ip4"
valid_status_codes: [200,404,403]
icmp:
prober: icmp
timeout: 10s
icmp:
preferred_ip_protocol: "ip4"
tcp_connect:
prober: tcp
timeout: 5s
serviceMonitor:
enabled: true
selfMonitor:
enabled: true
additionalMetricsRelabels: {}
additionalRelabeling: []
labels:
monitoring: prometheus
interval: 30s
scrapeTimeout: 30s
defaults:
labels:
release: prometheus-stack
additionalRelabeling:
- targetLabel: pod
replacement: blackbox-exporter-infra
targets:
- name: test
url: https://google.com
interval: 30s
scrapeTimeout: 5s
module: http_2xx
additionalMetricsRelabels:
module: http_2xx
environment: development

Enter the command that you execute and failing/misfunctioning.

helm install --kube-context test -n bbx bbx ./test_blackbox -f ./test_blackbox/tools-values.yaml

Anything else we need to know?

While deploying normally, it works well.
With the existing helm with argoCD setup, i can able to probe the targets manually.

curl bbx-prometheus-blackbox-exporter:9115/probe?target=https://google.com

ts=2023-10-16T18:07:26.377Z caller=handler.go:187 module=http_2xx target=https://google.com level=debug msg="Beginning probe" probe=http timeout_seconds=5
ts=2023-10-16T18:07:26.378Z caller=handler.go:187 module=http_2xx target=https://google.com level=debug msg="Resolving target address" target=www.google.com ip_protocol=ip4
ts=2023-10-16T18:07:26.438Z caller=handler.go:187 module=http_2xx target=https://google.com level=debug msg="Resolved target address" target=www.google.com ip=
ts=2023-10-16T18:07:26.439Z caller=handler.go:187 module=http_2xx target=https://google.com level=debug msg="Making HTTP request" url=https:// host=www.google.com
ts=2023-10-16T18:07:26.487Z caller=handler.go:187 module=http_2xx target=https://google.com level=debug msg="Received HTTP response" status_code=200
ts=2023-10-16T18:07:26.487Z caller=handler.go:187 module=http_2xx target=https://google.com level=debug msg="Response timings for roundtrip" roundtrip=0 start=2023-10-16T18:07:26.439173179Z dnsDone=2023-10-16T18:07:26.439173179Z connectDone=2023-10-16T18:07:26.442834394Z gotConn=2023-10-16T18:07:26.450037277Z responseStart=2023-10-16T18:07:26.487678402Z tlsStart=2023-10-16T18:07:26.442857592Z tlsDone=2023-10-16T18:07:26.450024645Z end=2023-10-16T18:07:26.487855879Z
ts=2023-10-16T18:07:26.488Z caller=handler.go:187 module=http_2xx target=https://google.com level=debug msg="Probe succeeded" duration_seconds=0.110171291

[nvtkaszpir]

probably caused by #3375