Add time based partitioning to store component #957
Conversation
|
CC @povilasv PTAL |
| @@ -145,6 +145,31 @@ func modelDuration(flags *kingpin.FlagClause) *model.Duration { | |||
| return value | |||
| } | |||
|
|
|||
| type flagTime struct { | |||
There was a problem hiding this comment.
so I belive this is quite fixed... We essentially need duration right? Like now-3months - now-2h style.
There was a problem hiding this comment.
Yeah I agree I need relative time also, would essentially replace the functionality I got here #930
There was a problem hiding this comment.
I think both would be useful. We're intending to use this for horizontal scale out, and we want the ability to specify exact time ranges in order to size the blocks served to the capacity of the thanos store host.
There was a problem hiding this comment.
👍
One question: I don't get why someone would need specific time. I think it does not make sense. What's the use case?
We're intending to use this for horizontal scale out, and we want the ability to specify exact time ranges in order to size the blocks served to the capacity of the thanos store host.
@claytono hmm are you sure you want fixed time ranges for that? Why?
There was a problem hiding this comment.
We intend to have a handful of Thanos store nodes, each serving a portion of a shared bucket, with one of them having an open ended time range for all new metrics. For now, we intend to have an outside process periodically do analysis of the bucket and generate time ranges for each thanos store process based on index size for each block. We want to provision these nodes such that they're all fairly full from a memory standpoint, but that we're not over-provisioning. For us, the major expense of running Thanos is the memory on compute instances, and the S3 storage is nearly free in comparison.
With metric ingest rates changing over time (new apps, seasonality, etc) and the activity of the compactor, I think partitioning the bucket time ranges with relative times is going to be error prone and/or lead to inefficient usage of the hardware.
There was a problem hiding this comment.
It's kind of odd from my perspective, but if you find this useful, sure (: happy to accept that.
There was a problem hiding this comment.
It's what we've come up with for horizontal scaling of the Thanos store nodes. I'd love to hear how other people are managing scaling out.
There was a problem hiding this comment.
For us, the major expense of running Thanos is the memory on compute instances
Seems, you are trying to solve separate problem with absolute time ranges.
I'd like to have relative time as well.
|
FYI I've continued work on different PR #1077 |
Changes
Add
--min-timeand--max-timecommand-line flags to store component. This will cause the store component to only load blocks whose start time lies within the time given. These two can be given independently and the defaults are effectively "forever ago" and "forever in the future" respectively.Verification
I've tested this against a copy of one of our production buckets and I've updated the e2e tests to test this functionality.
cc: #814