Sidecar: Added matchers support #3940

Namanl2001 · 2021-03-18T20:33:37Z

Fixes task 2: For Prometheus store (sidecar), add matchers to the LabelValues query of #3779

Changes

Implemented API /api/v1/status/buildinfo which returns the buildinfo of prometheus. Calling this API once when the sidecar starts and using it in pkg/store/prometheus.go labelValues method, if the version is > "2.24" used labelValues query otherwise fallback to series query.

Verification

Added unit tests

Signed-off-by: Namanl2001 <[email protected]>

pkg/promclient/promclient.go

pkg/store/prometheus.go

pkg/promclient/promclient.go

…ers-sidecar

Signed-off-by: Namanl2001 <[email protected]>

cmd/thanos/sidecar.go

pkg/store/prometheus.go

Signed-off-by: Namanl2001 <[email protected]>

yeya24

You still need to fix the failing test cases.

Signed-off-by: Namanl2001 <[email protected]>

pkg/store/prometheus.go

Signed-off-by: Namanl2001 <[email protected]>

yeya24 · 2021-04-14T15:30:36Z

cmd/thanos/sidecar.go

+						"err", err,
+					)
+					return err
+				}


I am still thinking about this logic.
This doesn't work well when Prometheus version is < 2.14.0, where /api/v1/status/buildinfo doesn't exist. In this case, we just get 404s but we are still retrying indefinitely. This seems wrong to me.

We can only retry for a short time or we check the return error to stop retrying.

Lets retry for lets say N times (maybe N=5) and exit if retry fails?

Also for Prometheus <2.14.0 this function would fail, so we also need to decide how to fetch build version for <2.14.0

As discussed in contributor hours:
Let’s check for 404, if it’s present, set 0 value for promVersion

cmd/thanos/sidecar.go

Signed-off-by: Namanl2001 <[email protected]>

yeya24 · 2021-04-17T03:51:47Z

cmd/thanos/sidecar.go

+			if err != nil {
+				return errors.Wrap(err, "failed to get prometheus version")
+			}
+			return nil


If you run this in a goroutine and it ends, the sidecar program also stops running because of the run group. Please make sure your code doesn't break the functionality. You can see the tests failed.

I think it is unnecessary to have a separate goroutine for checking the Prometheus version as it is unlikely to change.
We can run it in the main thread once? WDYT? @yashrsharma44 @Namanl2001 @bwplotka

I've got a doubt that we already have more than one actor in this run group but why there is no such problem with them?

I think it is unnecessary to have a separate goroutine for checking the Prometheus version as it is unlikely to change.

Do we add things in a separate goroutine which are likely to change?

We could put it somewhere here: https://github.com/thanos-io/thanos/pull/3940/files#diff-a89173b10deb4d469ed5524c1addf5b154030c04f1001f9dc91676b7966f1127R157

bwplotka

Some comments during Community call

bwplotka · 2021-04-22T17:27:18Z

cmd/thanos/sidecar.go

+			if err != nil {
+				return errors.Wrap(err, "failed to get prometheus version")
+			}
+			return nil


We could put it somewhere here: https://github.com/thanos-io/thanos/pull/3940/files#diff-a89173b10deb4d469ed5524c1addf5b154030c04f1001f9dc91676b7966f1127R157

bwplotka · 2021-04-22T17:28:45Z

pkg/store/prometheus.go

+	version := p.promVersion()
+
+	if len(r.Matchers) == 0 || version > "2.24" {
+		vals, err = p.client.LabelValuesInGRPC(ctx, p.base, r.Label, r.Matchers, r.Start, r.End)


Are we making gRPC call?

The naming makes me confused, too. We are making HTTP calls but return GRPC error.

pkg/store/prometheus.go

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 · 2021-04-23T21:51:17Z

As suggested, removed the separate goroutine for fetching the Prometheus version and moved the code block to an existing goroutine.

yeya24

LGTM. @Namanl2001 Thanks for your patience and great work!

yeya24 · 2021-04-26T17:32:59Z

Need another look for this pr to merge. Maybe @yashrsharma44 @bwplotka ?

yashrsharma44

I have added some nits regarding some edge cases when the version is not a valid string. If this case sounds unlikely, we can go ahead with the current implementation, looks good to me.

pkg/store/prometheus.go

pkg/store/prometheus_test.go

yashrsharma44 · 2021-04-27T08:35:11Z

pkg/store/prometheus.go

+	)
+
+	vals := []string{}
+	version := p.promVersion()


p.promVersion might return some garbage value if the m.BuildVersion doesn't run correctly, so let's make sure version is a correct version string, and not some garbage value.

pkg/store/prometheus_test.go

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 · 2021-04-27T21:17:50Z

PHAL! @yeya24 @yashrsharma44

yashrsharma44

Suggested some coding style, otherwise LGTM 🔥

yashrsharma44 · 2021-04-28T15:34:22Z

pkg/store/prometheus.go

+	version, err := semver.Parse(v)
+	if err == nil {
+		baseVer, err := semver.Make("2.24.0")
+		if err != nil {
+			return nil, err
+		}
+		if version.Compare(baseVer) == 1 {
+			lvc = true
+		}
+	}
+
+	if len(r.Matchers) == 0 || lvc {


Suggested change

version, err := semver.Parse(v)

if err == nil {

baseVer, err := semver.Make("2.24.0")

if err != nil {

return nil, err

}

if version.Compare(baseVer) == 1 {

lvc = true

}

}

if len(r.Matchers) == 0 || lvc {

// Make sure the prometheus version is a valid semver type.

version, err := semver.Parse(v)

if err != nil {

return nil, err

}

baseVer, err := semver.Make("2.24.0")

if err != nil {

return nil, err

}

if len(r.Matchers) == 0 || version.Compare(baseVer) == 1 {

Lets simplify things, return early if an error - https://thanos.io/tip/contributing/coding-style-guide.md/#control-structure-prefer-early-returns-and-avoid-else

I think if the version is malformed we can still make a series call instead of returning the error. Please see If it is malformed, we just use the series way in #3940 (comment)

yashrsharma44 · 2021-04-28T17:52:02Z

pkg/store/prometheus.go

+	version, err := semver.Parse(v)
+	if err == nil {
+		baseVer, err := semver.Make("2.24.0")
+		if err != nil {


If this is true(https://github.com/thanos-io/thanos/pull/3940/files#r622405182), then we should not exit early. Then we can remove the error check.

Then we can remove the error check.

Sorry, are you saying about the error check at Line 515? As it is for handling error retuned while initializing the baseVer.

Yup, if it errors out, we can revert to our original behaviour of -

we can still make a series call instead of returning the error

Does this look good?

version, err1 := semver.Parse(v) baseVer, err2 := semver.Make("2.24.0") if err1 == nil && err2 == nil && version.Compare(baseVer) == 1 { lvc = true }

Apologies for elongating this, but we can simplify things by using this package

I checked their docs and it stated for their Compare function, we can provide two strings, if the promVersion is invalid, the value would be 0, so we dont have to worry about that. So lets use this -

defaultVersion := "2.24.0" if len(r.Matchers) == 0 || semver.Compare(version, defaultVersion) == 1 {

Our current implementation is making it unnecessarily verbose and by using the above package, we can simplify things 😄

Any thoughts on this @yeya24? 🤗 As you earlier suggested to use a library #3940 (comment)

If we already use the semver library then let's use it here as well.

Signed-off-by: Namanl2001 <[email protected]>

yeya24 · 2021-04-29T18:18:00Z

pkg/store/prometheus.go

+	v := p.promVersion()
+
+	version, err1 := semver.Parse(v)
+	baseVer, err2 := semver.Make("2.24.0")


You can make the base version a global variable and we know it doesn't throw an error

Why should we make it a global variable? 🤔 We can still keep baseVer a local variable and don't check for an error while declaring and initializing it.

For performance. If you call LabelValues 1k times, then semver.Make("2.24.0") will be called 1k times. If you make it a global variable then it is parsed only once.

pkg/store/prometheus_test.go

Signed-off-by: Namanl2001 <[email protected]>

yeya24

LGTM

yeya24 · 2021-05-05T19:09:17Z

cc @yashrsharma44 @bwplotka for a final review. If there is no objection, I can merge it tomorrow.

yashrsharma44

LGTM 💪

Namanl2001 added 3 commits March 16, 2021 20:28

matchers support in sidecar

00e9eb2

Signed-off-by: Namanl2001 <[email protected]>

added and used buildinfo api

51a162d

Signed-off-by: Namanl2001 <[email protected]>

go formatting fix

f4096da

Signed-off-by: Namanl2001 <[email protected]>

yeya24 reviewed Mar 24, 2021

View reviewed changes

Namanl2001 added 2 commits March 27, 2021 22:12

Merge branch 'main' of https://github.com/thanos-io/thanos into match…

f4069b7

…ers-sidecar

addressed minor changes

6c2ec33

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 requested a review from yeya24 April 2, 2021 14:06

yeya24 reviewed Apr 2, 2021

View reviewed changes

cmd/thanos/sidecar.go Outdated Show resolved Hide resolved

pkg/store/prometheus.go Outdated Show resolved Hide resolved

pkg/store/prometheus.go Show resolved Hide resolved

using set to handle duplicate values

a4073e4

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 dismissed a stale review via a4073e4 April 3, 2021 18:07

Namanl2001 requested a review from yeya24 April 3, 2021 18:29

yeya24 reviewed Apr 3, 2021

View reviewed changes

Namanl2001 added 2 commits April 14, 2021 17:28

Logic correction

fe5067a

Signed-off-by: Namanl2001 <[email protected]>

test case update

92c3234

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 requested a review from yeya24 April 14, 2021 13:48

yeya24 reviewed Apr 14, 2021

View reviewed changes

pkg/store/prometheus.go Outdated Show resolved Hide resolved

changes addressed

df6b6ef

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 requested a review from yeya24 April 14, 2021 14:35

yeya24 reviewed Apr 14, 2021

View reviewed changes

yashrsharma44 reviewed Apr 15, 2021

View reviewed changes

cmd/thanos/sidecar.go Outdated Show resolved Hide resolved

handling promVersion < 2.14

aaf7906

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 requested review from yashrsharma44 and yeya24 April 16, 2021 18:53

yeya24 reviewed Apr 17, 2021

View reviewed changes

bwplotka reviewed Apr 22, 2021

View reviewed changes

removed goroutine for promVersion

ef797f6

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 requested a review from yeya24 April 23, 2021 21:53

yeya24 previously approved these changes Apr 23, 2021

View reviewed changes

yashrsharma44 suggested changes Apr 27, 2021

View reviewed changes

using semver

fd708fe

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 dismissed yeya24’s stale review via fd708fe April 27, 2021 18:41

making linter happy1

d874bc1

Signed-off-by: Namanl2001 <[email protected]>

yashrsharma44 suggested changes Apr 28, 2021

View reviewed changes

yashrsharma44 reviewed Apr 28, 2021

View reviewed changes

small nits

e59208c

Signed-off-by: Namanl2001 <[email protected]>

Namanl2001 requested review from yashrsharma44 and yeya24 April 29, 2021 14:36

yeya24 requested changes Apr 29, 2021

View reviewed changes

Namanl2001 added 2 commits April 30, 2021 02:12

made baseVer global

893e37c

Signed-off-by: Namanl2001 <[email protected]>

linting correction

92570d0

Signed-off-by: Namanl2001 <[email protected]>

yeya24 approved these changes Apr 29, 2021

View reviewed changes

yashrsharma44 approved these changes May 5, 2021

View reviewed changes

yeya24 merged commit 7aa6f97 into thanos-io:main May 7, 2021

Sidecar: Added matchers support #3940

Sidecar: Added matchers support #3940

Conversation

Namanl2001 commented Mar 18, 2021 • edited Loading

Changes

Verification

yeya24 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Namanl2001 Apr 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Namanl2001 commented Apr 23, 2021 • edited Loading

yeya24 left a comment

Choose a reason for hiding this comment

yeya24 commented Apr 26, 2021

yashrsharma44 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Namanl2001 commented Apr 27, 2021

yashrsharma44 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Namanl2001 Apr 28, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yeya24 Apr 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

yeya24 commented May 5, 2021

yashrsharma44 left a comment

Choose a reason for hiding this comment

Namanl2001 commented Mar 18, 2021 •

edited

Loading

Namanl2001 Apr 19, 2021 •

edited

Loading

Namanl2001 commented Apr 23, 2021 •

edited

Loading

Namanl2001 Apr 28, 2021 •

edited

Loading

yeya24 Apr 29, 2021 •

edited

Loading