Stateless Rule: Rollout - Alert For State is lost across restart - Blinking alerts

[ahurtaud]

Thanos, Prometheus and Golang version used:
thanos v0.25

What happened:

I implemented the stateless mode for the rule component.
When my rulers (k8s deployments) rollouts, every alerts already firing before the rollouts with a long for clause (lets say 30minutes) will be in Pending for that amount of time and firing again after 30minutes.

And because of our alertmanager config, alerts are resolved after 5minutes...

What you expected to happen:

I would like the alerts to still have a state for the for clause, maybe with ALERT_FOR_STATE metrics sent to the receivers?

Is it a valid issue? Is there anything I misunderstood / Do others have same issue?

Thanks

[yeya24]

If I understand correctly, in Prometheus, the state persistence across restart is by checking ALERTS_FOR_STATE time series. As mentioned by https://ganeshvernekar.com/blog/google-summer-of-code-2018-with-prometheus#1-persist-for-state-of-alerts-1.

I need to double check if this mode supports this by using the upstream rule manager. And we need more E2E tests to verify this.

[yeya24]

Yeah, I can confirm this is a bug, or a feature instead I would say. Working on the fix and tests now. We need to implement a Queryable and Querier for HTTP-based Thanos queries. Not sure if I can fix it this week.

Another way to go is to just implement remote read for Thanos Querier? Then we can just connect it using remote storage client.

[ahurtaud]

Yeah I know this will not be an "as easy as it looks" hence creating directly a github issue.
I think it is fair to agree that ALERT_FOR_STATE must be queryable through the same query endpoint.
(meaning stateless ruler MUST remote_write to thanos receive (or other) and register it in the query path of that ruler.)

From what I read from Ganesh, When ruler restarts, it queries ALERT_FOR_STATE for each alertname, and applies the same state if found:
something like that:

WDYT?

[yeya24]

Yeah, i think that's correct. Right now we just need to implement the Querier interface for Thanos Querier so that we can fetch that time series from it.

[yeya24]

Created prometheus/prometheus#10443 on upstream prometheus to address one of the issue. After that one is merged then I can open a new pr to fix this.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[ahurtaud]

referencing #5230 for implementation of the fix.

[yeya24]

I will close this one as the feature was merged into main already. Let us know how the feature works