Make store-gateway -querier.query-store-after aware

[pracucci]

Is your feature request related to a problem? Please describe.
The store-gateway sync blocks every -blocks-storage.bucket-store.sync-interval=15m. At each interval, it downloads the bucket index of each tenant and, for each tenant, loads new blocks and unload deleted blocks belonging to its shard.

This means that the store-gateway loads all 2h blocks uploaded by ingesters and then unload them once the compactor has run and these blocks have been compacted together in the 2h range. With many ingesters or tenants, this means quite a lot of work.

However, we configure querier with -querier.query-store-after=12h. It means querier never asks store-gateway to fetch any series until 12h ago. So, why loading all these 2h source blocks which compactor is likely to delete soon, before 12h passes?

Describe the solution you'd like
My proposal is to make store-gateway -querier.query-store-after aware. The idea is that store-gateway should not load at all any block containing samples within the last 12h (minus some grace period, minus the sync interval, but you got the point).

Additional context
After some offline discussion it came out a good feedback from @bboreham :

We sometimes change that setting to 0 - would we want a special feature in store-gateways to recognise this, or is restarting ok?

This is something we should consider we building this improvement. My feedback:

• The easy way is being aware of that and set it to 0 on store-gateways first and rollout queriers once store-gateways rollout is done.
• As an alternative, we could set it in the runtime config. However, if the setting is in the runtime config, then once you change it it applies to queriers and store-gateways nearly at the same time. Loading blocks in the store-gateway will take some time, so long-term queries will fail in the meanwhile because querier will not be able to query most recent blocks (not loaded on store-gateways yet). On the contrary, if we rollout the change on store-gateways first and then queriers, we don't have this issue.

[pstibrany]

#365 implements this idea, but introduces new flag. We don't want to use query-store-after directly, but want to load blocks earlier before queriers can start query them. Instead of computing this value automatically, we let operator configure it.