Staleness rules not remote writing ALERTS & ALERTS_FOR_STATE metric

[v1jayr]

I enabled remote write in thanos ruler and found that ALERTS & ALERTS_FOR_STATE metrics are missing in the remote store. I looked like PR was supposed to fix it and is part of v0.30.0 release. My Thanos ruler is v0.32.0 and still not seeing these 2 metrics. Any ideas what I might be missing?

[dosubot[bot]]

Hey @v1jayr! I'm here to help you with bugs, questions, and becoming a contributor. Let's squash those bugs together! 🤖

I found a similar closed issue that might be relevant to your situation. The issue was that in stateless mode, the ALERTS and ALERTS_FOR_STATE metrics were not being written because the alerts were never restored. This was due to the RestoreForState function using the fanout storage as queryable, which resulted in an error when the agent DB had no queryable. This prevented the rule from being restored, and consequently, the alert metrics were never appended. A related pull request (PR #5230) was mentioned as potentially resolving this problem ^[1].

Additionally, there is a test case in the Thanos codebase that demonstrates the remote writing of ALERTS metrics. The test case TestRule_CanRemoteWriteData in the test/e2e/rule_test.go file checks that Thanos Ruler can be run in stateless mode where it remote writes rule evaluations to a Prometheus remote-write endpoint ^[2].

If you are still experiencing issues with Thanos Ruler v0.32.0, it might be worth checking your configuration to ensure that everything is set up correctly for remote writing.

To continue talking to Dosu, mention @dosu.

[v1jayr]

@dosu I am running v0.32.0 which should include the fix you mention above and still I am not seeing the ALERTS metric. Does the fix require any additional flag to be enabled?

[dosubot[bot]]

I couldn't find any specific additional flags or configuration settings required to enable remote writing of ALERTS and ALERTS_FOR_STATE metrics in Thanos Ruler v0.32.0. However, it's important to ensure that your configuration aligns with the general requirements for remote write functionality. Also, be aware of the known limitations: the metadata_config is not supported in stateless mode, and the Store API won't be exposed if stateless mode is enabled ^[1].

To continue talking to Dosu, mention @dosu.

[GiedriusS]

See https://cloud-native.slack.com/archives/C167KFM6C/p1727715040364639?thread_ts=1727707085.019639&cid=C167KFM6C. Maybe this isn't true?

[v1jayr]

for reference here is my conversation Ben Ye who worked on a related commit https://cloud-native.slack.com/archives/CK5RSSC10/p1729199536051119