[query-frontend] Inconsistent results between querying frontend and querier directly

[verejoel]

Thanos, Prometheus and Golang version used: 0.32.3

Object Storage Provider: Azure

tl;dr: The bug in a nutshell

Deduped and non-deduped query responses seem to be different if you make them via the query frontend (QF) compared to directly in the querier (Q).

QF, deduped:

Q, deduped

QF, no dedupe

Q, no dedupe

Our story begins with the following query:

(max without (sloth_window) (slo:sli_error:ratio_rate5m{sloth_id="proxy-public-probes-latency",sloth_service="proxy",sloth_slo="public-probes-latency"} > (14.4 * 0.01)) and max without (sloth_window) (slo:sli_error:ratio_rate1h{sloth_id="proxy-public-probes-latency",sloth_service="proxy",sloth_slo="public-probes-latency"} > (14.4 * 0.01)))

It was reported that this query behaved strangely under deduplication. Without deduplication, 6 series were returned from the QF:

QF, no dedupe

``` {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} ```

The bug: When deduplication was selected (deduping on thanos_ruler_replica external label, no results were returned.

Executing the same query directly in the Q gives:

Q, no dedupe

``` {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11857", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11857", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11859", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11859", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12197", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12197", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} ```

Q, deduped

``` {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11857", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11859", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12197", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} ```

Deeper dive

Each side of the and statement seems to be deduped correctly, in both the Q and QF (i.e. the results below were the same in both components).

Left side, no dedupe

``` {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11857", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11857", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11859", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11859", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12197", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12197", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} ```

Left side, deduped

``` {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11857", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11859", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12197", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} ```

Right side, no dedupe

``` {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11857", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11857", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11859", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11859", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12197", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12197", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} ```

Right side, deduped

``` {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11857", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11859", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12197", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"} ```

First strange thing: there are clearly series here that I would expect to end up in the anded query. E.g.

# left side
{company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"}

# right side
{company_id="6", owner="websec", service_id="11856", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency"}

have the same labelsets, but do not appear in the 6 series returned from the QF.

I have tried rebooting the QF and Q to ensure that any caching is wiped clean between running queries. We do not use memcached or redis external caches.

Additional information

These data come from recording rules. In our current setup, we only have one external label on the ruler: thanos_ruler_replica. Because of this, when the compactor dedupes the data, it doesn't remove this label, it simply replaces it with the value deduped (see picture).

This behaviour is the same for instant queries. I evaluated the same query at the same evaluation time in the QF and the Q. Q gave me the full results deduped correctly. QF failed to dedupe and produced the reduced result set (6 series instead of 11).

What you expected to happen: Query frontend should give correct answers to queries!

How to reproduce it (as minimally and precisely as possible): In my setup, I was just querying the ruler. For testing, I used one QF and Q component. There were multiple stores/rulers tested, but using the store selector to limit the queries to one store also yielded the same behaviour.

It also occurred if I just used the store endpoint group:

Except in that case I got an additional label, deduped, from the compactor deduplication.

QF, no dedupe, stores

``` {company_id="3971", owner="websec", service_id="12426", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} | 14767.909965519248 -- | -- {company_id="3971", owner="websec", service_id="12428", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} | 7022.422099205556 {company_id="6", owner="websec", service_id="11858", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="deduped"} | 9588.259222207245 {company_id="6", owner="websec", service_id="11996", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} | 7451.229785714833 {company_id="6", owner="websec", service_id="12038", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} | 11231.732160469583 {company_id="6", owner="websec", service_id="12116", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="deduped"} | 11190.035731514101 {company_id="6", owner="websec", service_id="12373", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-1"} | 4091.483576870218 {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="deduped"} | 7335.330289910424 {company_id="6", owner="websec", service_id="12666", sloth_id="proxy-public-probes-latency", sloth_service="proxy", sloth_slo="public-probes-latency", thanos_ruler_replica="thanos-ruler-thanos-ruler-0"} | 7335.330289910424 ```

Full logs to relevant components: Issue reached the character limit, will post in comments.

[verejoel]

Querier Logs (nothing in QF)

direct to Q, no dedupe

``` ts=2023-10-04T09:21:43.196996081Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:21:43.197119077Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ```

direct to Q, deduped

``` ts=2023-10-04T09:22:37.562820467Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried" ts=2023-10-04T09:22:37.562860446Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ```

via QF, no dedupe

``` ts=2023-10-04T09:24:10.606817415Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:10.606862656Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ```

via QF, deduped

``` ts=2023-10-04T09:24:34.130692235Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.130704525Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.130782085Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.130843034Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.130990302Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.131033512Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.131101733Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.131207897Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried" ts=2023-10-04T09:24:34.139337994Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139397555Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139457017Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139492761Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139616304Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried" ts=2023-10-04T09:24:34.13971375Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139719972Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139719402Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139724458Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.13974456Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139744807Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.139798603Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139808176Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139720189Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.139713605Z caller=proxy.go:318 level=debug msg="Tenant info in Series()" tenant=default-tenant ts=2023-10-04T09:24:34.139910414Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.140010616Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.140010308Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried" ts=2023-10-04T09:24:34.1400055Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.140079053Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.140099719Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.140097017Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.140173182Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried;store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried" ts=2023-10-04T09:24:34.140232682Z caller=proxy.go:364 level=debug component=proxy request="min_time:1696377790000 max_time:1696378090000 matchers: matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_strategy:ABORT shard_info: without_replica_labels:\"prometheus_replica\" without_replica_labels:\"receive_replica\" without_replica_labels:\"tenant_id\" without_replica_labels:\"thanos_ruler_replica\" without_replica_labels:\"block_source\" " msg="Series: started fanout streams" status="store Addr: dns:///thanos-store-rule-shard-2.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{block_source=\"bucket-rewrite\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1665619204532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-0.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1664409604532 MaxTime: 1696406400000 queried;store Addr: dns:///thanos-store-rule-shard-1.thanos.svc:10901 LabelSets: {block_source=\"bucket-rewrite\", thanos_ruler_replica=\"deduped\"},{block_source=\"bucket-rewrite\", thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"deduped\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-0\"},{thanos_ruler_replica=\"thanos-ruler-thanos-ruler-1\"} MinTime: 1668038400518 MaxTime: 1696392000000 queried" ```

[verejoel]

Update: using the nightly build main-2023-10-04-5b7feeb seems to have resolved this issue! Probably thanks to #6605

[GiedriusS]

Closing this as per the last comment.