Store: selector labels from store #1059

jojohappy · 2019-04-19T03:48:30Z

Signed-off-by: jojohappy [email protected]

Changes

Implements: #1034

Add --selector labels for store gateway.

Verification

To see the store page to verify the announced labels field.

jojohappy · 2019-04-19T03:51:41Z

pkg/store/prometheus.go

+			return false, nil, nil
+		}
+
+		tms = append(tms[:i], tms[i+1:]...)


I have a question here. Why do we remove the match label from request LabelMatcher? I follow the implemete of the labelsMatches method above. But I could not catch the point :(

Sorry what do you mean remove? You mean continue part on line 315? This feels wrong. It's bit different to what we have here:

thanos/pkg/store/prometheus.go

Line 311 in d02488d

if extValue == "" {

That's why.. can we have just one function for matching + selector labels? (:

Re-reading the code, I'm sorry that I misunderstood the external_labels that those should not be in the storepb.LabelMatcher, because those would be added to any series. Here the selector-labels are as same as the external_labels used by store gateway sharding selector. We assume that all of series in the bucket queried by store component have the selector-labels pairs.

I have reused the matchesExternalLabels function to check selector-labels, and do translateMatcher again to get the new labels.Matcher for querying.

bwplotka · 2019-08-10T10:35:12Z

@jojohappy should we get back to this? Sorry for not reviewing this for some time - but make it non-draft - less chances the review will be postponed.

I think we all agree to this so let's implement this. Thanks for helping!

bwplotka

The direction is good - just some suggestions + can we rebase? (:

Thanks @jojohappy !

bwplotka · 2019-08-10T10:37:10Z

cmd/thanos/store.go

@@ -50,11 +51,20 @@ func registerStore(m map[string]setupFunc, app *kingpin.Application, name string
 	blockSyncConcurrency := cmd.Flag("block-sync-concurrency", "Number of goroutines to use when syncing blocks from object storage.").
 		Default("20").Int()

+	selectorLabels := cmd.Flag("selector-label", "Store Gateway selector labels that will be exposed in info endpoint (repeated).").


Let's make sure this is the same, consistent for receive and maybe rule? Querier either dropped it already or will drop as we aggregate labels there from storeAPIs.

bwplotka · 2019-08-10T10:40:17Z

pkg/store/prometheus.go

@@ -298,6 +298,32 @@ func labelsMatches(lset labels.Labels, ms []storepb.LabelMatcher) (bool, []store
 	return true, newMatcher, nil
 }

+func labelsMatchesAndTranslate(lset labels.Labels, ms []storepb.LabelMatcher) (bool, []labels.Matcher, error) {


This might work after few nits, but let's focus on reusing stuff. We do matching almost on every component. We for sure can figure out some helper/abstraction for this somehow critical path. (: More code reused = potentially fewer bugs and less code to read. What do you think?

Totally agree with you. I have delete the method and reuse the matchesExternalLabels to check the labels matcher.

bwplotka · 2019-08-10T10:48:39Z

pkg/store/prometheus.go

+			return false, nil, nil
+		}
+
+		tms = append(tms[:i], tms[i+1:]...)


Sorry what do you mean remove? You mean continue part on line 315? This feels wrong. It's bit different to what we have here:

thanos/pkg/store/prometheus.go

Line 311 in d02488d

if extValue == "" {

That's why.. can we have just one function for matching + selector labels? (:

jojohappy · 2019-08-12T05:34:18Z

@bwplotka I have updated. PTAL! After that, I will try to add some testings.

bwplotka

It looks good so far!

Thanks for doing this @jojohappy. Super simple and nice.

There is an important item in Querier we need to look carefully and adjust, please see the comment below.

bwplotka · 2019-08-21T06:20:15Z

CHANGELOG.md

@@ -15,6 +15,7 @@ We use *breaking* word for marking changes that are not backward compatible (rel

 - [#1358](https://github.com/thanos-io/thanos/pull/1358) Added `part_size` configuration option for HTTP multipart requests minimum part size for S3 storage type
 - [#1363](https://github.com/thanos-io/thanos/pull/1363) Thanos Receive now exposes `thanos_receive_hashring_nodes` and `thanos_receive_hashring_tenants` metrics to monitor status of hash-rings
+- [#1059](https://github.com/improbable-eng/thanos/pull/1059) Store: selector labels from store.


Suggested change

- [#1059](https://github.com/improbable-eng/thanos/pull/1059) Store: selector labels from store.

- [#1059](https://github.com/improbable-eng/thanos/pull/1059) Store: Added label selecting. This allows store gateway to serve only certain TSDB blocks from object store.

bwplotka · 2019-08-21T06:21:44Z

cmd/thanos/store.go

@@ -49,7 +50,15 @@ func registerStore(m map[string]setupFunc, app *kingpin.Application, name string
 	blockSyncConcurrency := cmd.Flag("block-sync-concurrency", "Number of goroutines to use when syncing blocks from object storage.").
 		Default("20").Int()

+	selectorLabels := cmd.Flag("store.selector-label", "Store Gateway selector labels that will be exposed in info endpoint (repeated).").


hm we have

selectorLabels := cmd.Flag("selector-label", "Query selector labels that will be exposed in info endpoint (repeated)."). PlaceHolder("<name>=\"<value>\"").Strings()

Maybe worth to be consistent with naming and remove store prefix? essentially it has not much to do with store API for example.

bwplotka · 2019-08-21T06:28:27Z

pkg/store/bucket.go

+	}
+
+	for _, lset := range s.selectorLabels {
+		infoResp.Labels = append(infoResp.Labels, storepb.Label{Name: lset.Name, Value: lset.Value})


Before this change we need to adjust Querier logic:

thanos/pkg/query/storeset.go

Line 248 in 865d5ec

// Check if the store has some external labels specified and if any if there are duplicates.

Otherwise, sidecar + store gateway will be assumed a "duplicate" in Querier. Let's think about how we can fix this.

The storeAPI duplication detection logic is embedded in Querier to make sure user doesn't put many sidecars that upload blocks with the same external labels (common bug). I think we might want to change storeset behavior to look on StoreAPI InfoResponse.StoreType

I only don't like the fact that for Querier all of this is no longer some StoreAPI but actually Sidecar, Store...

I might think we actually would need to change StoreType to Producer, Browser as that would be a good abstraction over different implementations.

Thoughts @povilasv @GiedriusS @brancz ?

Am I missing something or shouldn't this just be the set of all labelsets seen in all blocks?

Can you elaborate?

Or better: Let's maybe quickly write up some proposal as we want consistent thing everywhere we do selectors + maybe selecting prefixes as well.

cc @jojohappy

Signed-off-by: jojohappy <[email protected]>

bwplotka · 2019-09-04T12:25:34Z

I think we need this, but we need something we discussed here: #1245 (comment)

I think we need short proposal maybe (?)

bwplotka · 2019-09-06T08:40:43Z

@jojohappy are you happy to close this in the meantime to stop confusion? I will start some PR with the design it would be also awesome if you would join us in making this if you want (:

jojohappy · 2019-09-06T09:16:22Z

are you happy to close this in the meantime to stop confusion?

That's OK.

I will start some PR with the design it would be also awesome if you would join us in making this if you want (:

Yes, I'd like to.

bwplotka · 2019-09-07T03:02:52Z

#1501

bwplotka · 2019-09-19T08:40:26Z

cmd/thanos/store.go

@@ -56,12 +57,20 @@ func registerStore(m map[string]setupFunc, app *kingpin.Application, name string
 	maxTime := model.TimeOrDuration(cmd.Flag("max-time", "End of time range limit to serve. Thanos Store serves only blocks, which happened eariler than this value. Option can be a constant time in RFC3339 format or time duration relative to current time, such as -1d or 2h45m. Valid duration units are ms, s, m, h, d, w, y.").
 		Default("9999-12-31T23:59:59Z"))

+	selectorLabels := cmd.Flag("selector-label", "Query selector labels that will be exposed in info endpoint (repeated).").


On each component that works on the object storage (e.g Store GW and Compactor), add --selector.relabel-config (and corresponding --selector.relabel-config-file) as in proposal

On top of this we want to make sure Info() endpoint has exact labels of blocks that we shard.

jojohappy commented Apr 19, 2019

View reviewed changes

jojohappy force-pushed the feature/store_selector branch from 5215e60 to 877ff79 Compare April 19, 2019 09:16

jojohappy changed the title ~~Feature: selector labels from store~~ Store: selector labels from store Apr 24, 2019

jojohappy force-pushed the feature/store_selector branch from 877ff79 to 9707e55 Compare April 24, 2019 07:07

bwplotka self-requested a review August 10, 2019 10:33

bwplotka reviewed Aug 10, 2019

View reviewed changes

jojohappy force-pushed the feature/store_selector branch from 9707e55 to 633dfc3 Compare August 12, 2019 02:44

jojohappy marked this pull request as ready for review August 12, 2019 03:07

jojohappy force-pushed the feature/store_selector branch from 9728f6d to c8c2559 Compare August 21, 2019 02:48

bwplotka reviewed Aug 21, 2019

View reviewed changes

bwplotka mentioned this pull request Aug 27, 2019

Enable sharding compact instances by prometheus shard #1245

Closed

jojohappy added 7 commits August 28, 2019 17:33

Feature: selector labels from store

fcc5c2e

Signed-off-by: jojohappy <[email protected]>

Add CHANGELOG

ed59ed5

Signed-off-by: jojohappy <[email protected]>

Fixed undefined variable

23d2d8a

Signed-off-by: jojohappy <[email protected]>

Re-use existed function instead of adding new one

ce4fe69

Signed-off-by: jojohappy <[email protected]>

Addressed review comments && Add test case

a8b9ba9

Signed-off-by: jojohappy <[email protected]>

Update docs

284e6b9

Signed-off-by: jojohappy <[email protected]>

Rebase && Fixed test case

5cec0a2

Signed-off-by: jojohappy <[email protected]>

jojohappy force-pushed the feature/store_selector branch from 82734c3 to 5cec0a2 Compare August 28, 2019 09:49

bwplotka closed this Sep 6, 2019

jojohappy deleted the feature/store_selector branch September 9, 2019 10:19

bwplotka reviewed Sep 19, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store: selector labels from store #1059

Store: selector labels from store #1059

jojohappy commented Apr 19, 2019

jojohappy Apr 19, 2019 •

edited

Loading

bwplotka Aug 10, 2019

jojohappy Aug 12, 2019 •

edited

Loading

bwplotka commented Aug 10, 2019

bwplotka left a comment

bwplotka Aug 10, 2019

bwplotka Aug 10, 2019

jojohappy Aug 12, 2019

bwplotka Aug 10, 2019

jojohappy commented Aug 12, 2019

bwplotka left a comment

bwplotka Aug 21, 2019

bwplotka Aug 21, 2019

bwplotka Aug 21, 2019

brancz Aug 28, 2019

bwplotka Sep 4, 2019

bwplotka commented Sep 4, 2019

bwplotka commented Sep 6, 2019

jojohappy commented Sep 6, 2019

bwplotka commented Sep 7, 2019

bwplotka Sep 19, 2019

bwplotka Sep 19, 2019

	- [#1059](https://github.com/improbable-eng/thanos/pull/1059) Store: selector labels from store.
	- [#1059](https://github.com/improbable-eng/thanos/pull/1059) Store: Added label selecting. This allows store gateway to serve only certain TSDB blocks from object store.

Store: selector labels from store #1059

Store: selector labels from store #1059

Conversation

jojohappy commented Apr 19, 2019

Changes

Verification

jojohappy Apr 19, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jojohappy Aug 12, 2019 • edited Loading

Choose a reason for hiding this comment

bwplotka commented Aug 10, 2019

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jojohappy commented Aug 12, 2019

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka commented Sep 4, 2019

bwplotka commented Sep 6, 2019

jojohappy commented Sep 6, 2019

bwplotka commented Sep 7, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jojohappy Apr 19, 2019 •

edited

Loading

jojohappy Aug 12, 2019 •

edited

Loading