You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Restarted Prometheus, it started replaying WAL. Sidecar still reported its status as "READY" via /-/ready hence it appears in Thanos Query, and then Thanos Query starts lagging unnecessarily waiting for an answer from Prometheus.
By definition, Sidecar is not ready to serve traffic if Prometheus is down.
What you expected to happen:
For Sidecar to either not report the status as READY or for Sidecar/Query not to hang waiting for an answer from Prometheus.
How to reproduce it (as minimally and precisely as possible):
Restart bigger Prometheus
Inspect Sidecar status
Full logs to relevant components:
{"caller":"sidecar.go:207","err":"expected 2xx response, got 503. Body: Service Unavailable","level":"warn","msg":"heartbeat failed","ts":"2021-12-02T11:57:36.068816651Z"}
The text was updated successfully, but these errors were encountered:
Thanos, Prometheus and Golang version used:
0.23.1
Object Storage Provider:
N/A
What happened:
Restarted Prometheus, it started replaying WAL. Sidecar still reported its status as "READY" via /-/ready hence it appears in Thanos Query, and then Thanos Query starts lagging unnecessarily waiting for an answer from Prometheus.
By definition, Sidecar is not ready to serve traffic if Prometheus is down.
What you expected to happen:
For Sidecar to either not report the status as READY or for Sidecar/Query not to hang waiting for an answer from Prometheus.
How to reproduce it (as minimally and precisely as possible):
Full logs to relevant components:
The text was updated successfully, but these errors were encountered: