-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Liveness and readiness probes are failing with 503 in compactor and prometheus after upgrading to 0.17.0 #3466
Comments
The problem is gone when reverted to 0.16.0 |
When running |
I see the same issues with using thanos store in v0.16, did not see in v0.15 |
We have the same issue I think. bucket config: Scality S3 Fails on 0.17.1 and works on 0.16.0 |
dupe of #3395 ? |
Yup, that looks like the same issue. Thank you all for reporting this, let's fix and release 0.17.2 (: Also commented on the mentioned issue. Closing this one as dup 🤗 |
@bwplotka I am seeing the same behavior with this issue in thanos-store with an upgrade to 0.17 as well, while not the same component - pods will be end up redeploying after liveness checks fail for no apparent reason. |
Thanos, Prometheus and Golang version used:
Thanos 0.17.0, prometheus 2.22.2
Object Storage Provider:
ceph S3
What happened:
After upgrading thanos to 0.17.0, the liveness probes in prometheus and thanos-compactor pods start failing with 503 error, which causes pods to restart
What you expected to happen:
Pods keep running
How to reproduce it (as minimally and precisely as possible):
Not sure what triggered it, I simply upgraded a working installation
Full logs to relevant components:
Couldn't get any error in logs since container simply restarts, no errors logged
The text was updated successfully, but these errors were encountered: