mixin: Use 2m interval for Thanos sidecar healthcheck

During prometheus updates the alert is firing because the metric is initialized with a value of '0' before the first heartbeat is sent. As such, the evaluation of the alert results into actually taking just the value of time() into consideration and the alert fires with misleading information. Lets add a small delay of 2 minutes to the alert to ensure that a heartbeat has been sent in the meantime. Signed-off-by: Markos Chandras <[email protected]>
thanos-io · Sep 22, 2020 · e097018 · e097018
1 parent 7b5c426
commit e097018
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -62,6 +62,7 @@ Thanos Rule's `/api/v1/rules` endpoint no longer returns the old, deprecated `pa
 - [#3095](https://github.com/thanos-io/thanos/pull/3095) Ruler: Update the manager when all rule files are removed.
 - [#3105](https://github.com/thanos-io/thanos/pull/3105) Querier: Fix overwriting `maxSourceResolution` when auto downsampling is enabled.
 - [#3010](https://github.com/thanos-io/thanos/pull/3010) Querier: Added `--query.lookback-delta` flag to override the default lookback delta in PromQL. The flag should be lookback delta should be set to at least 2 times of the slowest scrape interval. If unset it will use the PromQL default of 5m.
+- [#3204](https://github.com/thanos-io/thanos/pull/3204) Mixin: Use 2m interval for Thanos sidecar healthcheck.
 
 ### Added
 

diff --git a/mixin/alerts/sidecar.libsonnet b/mixin/alerts/sidecar.libsonnet
@@ -31,6 +31,7 @@
             expr: |||
               time() - max(thanos_sidecar_last_heartbeat_success_time_seconds{%(selector)s}) by (job, pod) >= 600
             ||| % thanos.sidecar,
+            'for': '2m'
             labels: {
               severity: 'critical',
             },