HA handling for store nodes

[fabxc]

Store nodes are currently generally run as a single replica. It's not super critical to have HA in general since several hours or even days of recent data are HA via the Prometheus servers. But for some scenarios it might still be preferable.

Two could simply be deployed and the query node would take care of deduplication/merging just like for Prometheus HA pairs. But unlike Prometheus servers, the underlying data is truly the same in this case and fetching twice the amount is unnecessary overhead.

Some simple logic could be added to the query node to recognize real duplicates (Prometheus HA pairs are actually different through a replica label) and to only query one of them.

[bwplotka]

I think we might need that sooner than later... (: How can we do it easily? Basically we need to tell querier that these X stores are the same thing.. Can we reuse labels field from store Info endpoint?

[dupondje]

The most basic way would just be the option to add for example --bucketid="xxx" to the storage command.
If the query command notices 2 (or more) buckets with the same ID, it could just take a random one to get its data from instead of all of them.

[deejay1]

For active/passive this could be done using a leader latch protocol and sharing the data downloaded by the leader as it could announce any new downloaded bucket via gossip (for a faster failover) and share it via HTTP/gRPC. This would eliminate the need to fetch the data from an object store directly and allow for the query nodes to have only a single source of truth (the current leader)

[mattbostock]

I'd like to volunteer to take this on. For our use case, downtime caused by the store instance fronting an S3 bucket being rescheduled to another machine is not really palatable.

I'm thinking of an active-active solution, since it avoids some of the complexities around deciding which instance is 'active' and would be more efficient with resources. As store nodes are essentially just caches, I think it should reasonable straightforward to achieve.

While thinking about high availability, we should also consider allowing the store nodes to scale horizontally for very large deployments, effectively allowing horizontal scaling the LRU cache of indices.

I propose:

• We shard the index cache across multiple store instances.
• Optionally, we replicate the shards to provide high availability for a single shard - though by having multiple shards, we can already improve overall availability and reduce the time to recovery.

Just an idea: If we have multiple shards, we might simplify the store instances by avoiding persisting the cache to disk, since the amount of data to pull from object storage would be reduced by 1/n where n is the number of shards when restarting a singe instance (though recovery would be slow if all instances are restarted).

[bwplotka]

@mattbostock Thanks!

It all works for one assumption: Thanos setup has only bucket to take data from, are we ok with it? I have seen some use cases for multiple buckets connected to same Thanos "cluster/network/setup", because "it is easier to manage", "my object storage is specific" etc. Maybe that's separate issue, but woth to be aware of this while implementing HA.

We shard the index cache across multiple store instances.

Makes sense, just I would love to hear/see more about the implementation details. As you suggested offline: https://godoc.org/github.com/golang/groupcache sound nice but it means that we are talking about sharding fully on stores (you ask whatever store and it gives you correct answer 100% time even if it needs to ask its peers) or maybe we want thanos-query to be aware of store sharding? Also are we are talking about sharding index cache based on... what? On matchers 0.o? __name__ only? what if someone asks for __name__~=.*?

though by having multiple shards, we can already improve overall availability and reduce the time to recovery.

Totally agree and thanks for example 👍 However, I would start from something simple first - just replicating (so true HA), because that is what you need (from you what you say). This will enable horizontal scaling (will offload single store) and potentially improve performance as well. Just sharding will ONLY improve the availability (but will still have some major disruption time), regarding the performance it is hard to say without #346 (which is in progress).

[mattbostock]

Added a proposal for high-availability for store instances here: #404

[bwplotka]

This can be solved by just by running multiple of Store Gateways behind any Loadbalancer (like Kuberentes Service) and without gossip.