Sidecar "ready" when Prometheus is replaying WAL

[GiedriusS]

Thanos, Prometheus and Golang version used:

0.23.1

Object Storage Provider:

N/A

What happened:

Restarted Prometheus, it started replaying WAL. Sidecar still reported its status as "READY" via /-/ready hence it appears in Thanos Query, and then Thanos Query starts lagging unnecessarily waiting for an answer from Prometheus.

By definition, Sidecar is not ready to serve traffic if Prometheus is down.

What you expected to happen:

For Sidecar to either not report the status as READY or for Sidecar/Query not to hang waiting for an answer from Prometheus.

How to reproduce it (as minimally and precisely as possible):

• Restart bigger Prometheus
• Inspect Sidecar status

Full logs to relevant components:

{"caller":"sidecar.go:207","err":"expected 2xx response, got 503. Body: Service Unavailable","level":"warn","msg":"heartbeat failed","ts":"2021-12-02T11:57:36.068816651Z"}

[GiedriusS]

Fixed in #4939.