Store-gateway: high memory allocations caused by per-tenant Prometheus registry

[grafanabot]

Describe the bug
To be able to use Thanos BucketStore while supporting Cortex multi-tenancy we need to create a BucketStore for each tenant, passing a dedicated Prometheus registry to each one and then aggregate metrics from all registries.

Due to this, the Prometheus metrics collection causes high memory allocations (order of 50MB/s in a store-gateway with 7.5K tenants). Allocated memory is not retained, but still puts pressure on GC.

In a cluster with low QPS, 95% store-gateway memory allocations are caused by metrics collecting.

Submitted by: pracucci
Cortex Issue Number: 3697

[grafanabot]

Enabling shuffle-sharding on store-gateway significantly improve this.

Submitted by: pracucci

[pracucci]

More data points from a store-gateway loading blocks from 13k tenants.

CPU

Memory allocations (bytes)