Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query: add query metrics to calls going through the Store API #5741

Merged
merged 37 commits into from
Oct 18, 2022

Conversation

douglascamata
Copy link
Contributor

@douglascamata douglascamata commented Sep 29, 2022

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Overall, this pull request's intent is to implement the first step in the proposal More granular query performance metrics.

Verification

  • Built the containers and ran it locally.
  • Added some tests, including e2e tests.

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
@douglascamata douglascamata requested review from jessicalins and matej-g and removed request for matej-g and jessicalins October 10, 2022 12:44
@douglascamata
Copy link
Contributor Author

I wanted to re-request review from the 3 reviewers, but Github is drunk. Sorry for the possible notification spam.

image

jessicalins
jessicalins previously approved these changes Oct 13, 2022
Copy link
Contributor

@jessicalins jessicalins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks 💪🏽 (:

pkg/store/telemetry/telemetry.go Outdated Show resolved Hide resolved
Comment on lines 59 to 60
"series_le": strconv.Itoa(int(seriesLeBucket)),
"samples_le": strconv.Itoa(int(samplesLeBucket)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concerns is still this: Is such "weird" metric usable in PromQL? (two dimensional histogram)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed in #5741 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it's actually len(series_le) * len(samples_le) number of histograms.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I am fine then, thanks for explanation (:

bwplotka
bwplotka previously approved these changes Oct 17, 2022
Comment on lines 59 to 60
"series_le": strconv.Itoa(int(seriesLeBucket)),
"samples_le": strconv.Itoa(int(samplesLeBucket)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it's actually len(series_le) * len(samples_le) number of histograms.

Comment on lines 59 to 60
"series_le": strconv.Itoa(int(seriesLeBucket)),
"samples_le": strconv.Itoa(int(samplesLeBucket)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I am fine then, thanks for explanation (:

matej-g
matej-g previously approved these changes Oct 17, 2022
Copy link
Collaborator

@matej-g matej-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(previously approved, approving after merging main and merging this PR now)

@matej-g matej-g merged commit be95bb1 into thanos-io:main Oct 18, 2022
utukJ pushed a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
…-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <[email protected]>

* Fix some linter warnings

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless logs

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <[email protected]>

* Remove TODO comment

Signed-off-by: Douglas Camata <[email protected]>

* Fix query tests

Signed-off-by: Douglas Camata <[email protected]>

* Reformat query docs

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless return

Signed-off-by: Douglas Camata <[email protected]>

* Put back old query docs

Signed-off-by: Douglas Camata <[email protected]>

* Update query docs again

Signed-off-by: Douglas Camata <[email protected]>

* Fix e2e env name

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <[email protected]>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <[email protected]>

* Clean up test

Signed-off-by: Douglas Camata <[email protected]>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <[email protected]>

* Remove extra empty line

Signed-off-by: Douglas Camata <[email protected]>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <[email protected]>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <[email protected]>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: utukj <[email protected]>
utukJ added a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
utukJ added a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
utukJ pushed a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
…-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <[email protected]>

* Fix some linter warnings

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless logs

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <[email protected]>

* Remove TODO comment

Signed-off-by: Douglas Camata <[email protected]>

* Fix query tests

Signed-off-by: Douglas Camata <[email protected]>

* Reformat query docs

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless return

Signed-off-by: Douglas Camata <[email protected]>

* Put back old query docs

Signed-off-by: Douglas Camata <[email protected]>

* Update query docs again

Signed-off-by: Douglas Camata <[email protected]>

* Fix e2e env name

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <[email protected]>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <[email protected]>

* Clean up test

Signed-off-by: Douglas Camata <[email protected]>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <[email protected]>

* Remove extra empty line

Signed-off-by: Douglas Camata <[email protected]>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <[email protected]>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <[email protected]>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: utukj <[email protected]>
utukJ added a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
GiedriusS added a commit that referenced this pull request Oct 27, 2022
…specify metric labels to collect (#5785)

* added unit tests for long labels and no external labels

Signed-off-by: utukj <[email protected]>

* trimmed too long external labels

Signed-off-by: utukj <[email protected]>

* added optional label selection

Signed-off-by: utukj <[email protected]>

* added cmd flag for choosing metric labels

Signed-off-by: utukj <[email protected]>

* updated docs

Signed-off-by: utukj <[email protected]>

* Update pkg/query/endpointset.go

full sentence fix

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* minor fixes from code review

Signed-off-by: utukj <[email protected]>

* fixed code comments

Signed-off-by: utukj <[email protected]>

* used enum for labels

Signed-off-by: utukj <[email protected]>

* updated query docs

Signed-off-by: utukj <[email protected]>

* cleaned up tests

Signed-off-by: utukj <[email protected]>

* Updates busybox SHA (#5793)

Signed-off-by: GitHub <[email protected]>

Signed-off-by: GitHub <[email protected]>
Co-authored-by: yeya24 <[email protected]>
Signed-off-by: utukj <[email protected]>

* Receive: Reload tenant limit configuration on file change (#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <[email protected]>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <[email protected]>

* Run goimports

Signed-off-by: Douglas Camata <[email protected]>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <[email protected]>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <[email protected]>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <[email protected]>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <[email protected]>

* Update lastReload

Signed-off-by: Douglas Camata <[email protected]>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <[email protected]>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <[email protected]>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <[email protected]>

* Run tests in parallel

Signed-off-by: Douglas Camata <[email protected]>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <[email protected]>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <[email protected]>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <[email protected]>

* Use absolute path to config file

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of parallel test

Signed-off-by: Douglas Camata <[email protected]>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless sleep

Signed-off-by: Douglas Camata <[email protected]>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <[email protected]>

* Remove unused fucntion

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <[email protected]>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <[email protected]>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <[email protected]>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <[email protected]>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <[email protected]>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <[email protected]>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: utukj <[email protected]>

* Query: add query metrics to calls going through the Store API (#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <[email protected]>

* Fix some linter warnings

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless logs

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <[email protected]>

* Remove TODO comment

Signed-off-by: Douglas Camata <[email protected]>

* Fix query tests

Signed-off-by: Douglas Camata <[email protected]>

* Reformat query docs

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless return

Signed-off-by: Douglas Camata <[email protected]>

* Put back old query docs

Signed-off-by: Douglas Camata <[email protected]>

* Update query docs again

Signed-off-by: Douglas Camata <[email protected]>

* Fix e2e env name

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <[email protected]>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <[email protected]>

* Clean up test

Signed-off-by: Douglas Camata <[email protected]>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <[email protected]>

* Remove extra empty line

Signed-off-by: Douglas Camata <[email protected]>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <[email protected]>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <[email protected]>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: utukj <[email protected]>

* docs: mark me as shepherd for next release (#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: utukj <[email protected]>

* Revert "docs: mark me as shepherd for next release (#5797)"

This reverts commit ea646a6.

Signed-off-by: utukj <[email protected]>

* Revert "Query: add query metrics to calls going through the Store API (#5741)"

This reverts commit eec4fd0.

Signed-off-by: utukj <[email protected]>

* Revert "Receive: Reload tenant limit configuration on file change (#5673)"

This reverts commit 24e1cc0.

Signed-off-by: utukj <[email protected]>

* Revert "Updates busybox SHA (#5793)"

This reverts commit 9474c00.

Signed-off-by: utukj <[email protected]>

* Updates busybox SHA (#5793)

Signed-off-by: GitHub <[email protected]>

Signed-off-by: GitHub <[email protected]>
Co-authored-by: yeya24 <[email protected]>
Signed-off-by: utukj <[email protected]>

* Receive: Reload tenant limit configuration on file change (#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <[email protected]>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <[email protected]>

* Run goimports

Signed-off-by: Douglas Camata <[email protected]>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <[email protected]>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <[email protected]>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <[email protected]>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <[email protected]>

* Update lastReload

Signed-off-by: Douglas Camata <[email protected]>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <[email protected]>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <[email protected]>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <[email protected]>

* Run tests in parallel

Signed-off-by: Douglas Camata <[email protected]>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <[email protected]>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <[email protected]>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <[email protected]>

* Use absolute path to config file

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of parallel test

Signed-off-by: Douglas Camata <[email protected]>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless sleep

Signed-off-by: Douglas Camata <[email protected]>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <[email protected]>

* Remove unused fucntion

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <[email protected]>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <[email protected]>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <[email protected]>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <[email protected]>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <[email protected]>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <[email protected]>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: utukj <[email protected]>

* Query: add query metrics to calls going through the Store API (#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <[email protected]>

* Fix some linter warnings

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless logs

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <[email protected]>

* Remove TODO comment

Signed-off-by: Douglas Camata <[email protected]>

* Fix query tests

Signed-off-by: Douglas Camata <[email protected]>

* Reformat query docs

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless return

Signed-off-by: Douglas Camata <[email protected]>

* Put back old query docs

Signed-off-by: Douglas Camata <[email protected]>

* Update query docs again

Signed-off-by: Douglas Camata <[email protected]>

* Fix e2e env name

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <[email protected]>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <[email protected]>

* Clean up test

Signed-off-by: Douglas Camata <[email protected]>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <[email protected]>

* Remove extra empty line

Signed-off-by: Douglas Camata <[email protected]>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <[email protected]>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <[email protected]>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: utukj <[email protected]>

* docs: mark me as shepherd for next release (#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: utukj <[email protected]>

* Revert "docs: mark me as shepherd for next release (#5797)"

This reverts commit c509c0e.

Signed-off-by: utukj <[email protected]>

* Revert "Updates busybox SHA (#5793)"

This reverts commit ad11a03.

Signed-off-by: utukj <[email protected]>

* Revert "Query: add query metrics to calls going through the Store API (#5741)"

This reverts commit 7a77769.

Signed-off-by: utukj <[email protected]>

* Revert "Receive: Reload tenant limit configuration on file change (#5673)"

This reverts commit 32ca327.

Signed-off-by: utukj <[email protected]>

* fixed lint issue

Signed-off-by: utukj <[email protected]>

* added unit test for truncate and clean up

Signed-off-by: utukj <[email protected]>

* fixed truncate label func and added more tests

Signed-off-by: utukj <[email protected]>

* removed name from truncate test

Signed-off-by: utukj <[email protected]>

* reorganized test cases and removed redundant comments

Signed-off-by: utukj <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* fixed failing checks

Signed-off-by: utukj <[email protected]>

* e2e: Adding test for querier with two stores loadbalancing across them.

Signed-off-by: bwplotka <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* dumped long expected output in unittest

Signed-off-by: utukj <[email protected]>

* Revert "e2e: Adding test for querier with two stores loadbalancing across them."

This reverts commit 96b1545.

Signed-off-by: utukj <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* moved label definition to endpointset

Signed-off-by: utukj <[email protected]>

Signed-off-by: utukj <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>
Signed-off-by: GitHub <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: bwplotka <[email protected]>
Co-authored-by: Bartlomiej Plotka <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: yeya24 <[email protected]>
Co-authored-by: Douglas Camata <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Co-authored-by: Giedrius Statkevičius <[email protected]>
GiedriusS added a commit to vinted/thanos that referenced this pull request Oct 31, 2022
…specify metric labels to collect (thanos-io#5785)

* added unit tests for long labels and no external labels

Signed-off-by: utukj <[email protected]>

* trimmed too long external labels

Signed-off-by: utukj <[email protected]>

* added optional label selection

Signed-off-by: utukj <[email protected]>

* added cmd flag for choosing metric labels

Signed-off-by: utukj <[email protected]>

* updated docs

Signed-off-by: utukj <[email protected]>

* Update pkg/query/endpointset.go

full sentence fix

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* minor fixes from code review

Signed-off-by: utukj <[email protected]>

* fixed code comments

Signed-off-by: utukj <[email protected]>

* used enum for labels

Signed-off-by: utukj <[email protected]>

* updated query docs

Signed-off-by: utukj <[email protected]>

* cleaned up tests

Signed-off-by: utukj <[email protected]>

* Updates busybox SHA (thanos-io#5793)

Signed-off-by: GitHub <[email protected]>

Signed-off-by: GitHub <[email protected]>
Co-authored-by: yeya24 <[email protected]>
Signed-off-by: utukj <[email protected]>

* Receive: Reload tenant limit configuration on file change (thanos-io#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <[email protected]>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <[email protected]>

* Run goimports

Signed-off-by: Douglas Camata <[email protected]>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <[email protected]>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <[email protected]>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <[email protected]>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <[email protected]>

* Update lastReload

Signed-off-by: Douglas Camata <[email protected]>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <[email protected]>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <[email protected]>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <[email protected]>

* Run tests in parallel

Signed-off-by: Douglas Camata <[email protected]>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <[email protected]>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <[email protected]>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <[email protected]>

* Use absolute path to config file

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of parallel test

Signed-off-by: Douglas Camata <[email protected]>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless sleep

Signed-off-by: Douglas Camata <[email protected]>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <[email protected]>

* Remove unused fucntion

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <[email protected]>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <[email protected]>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <[email protected]>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <[email protected]>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <[email protected]>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <[email protected]>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: utukj <[email protected]>

* Query: add query metrics to calls going through the Store API (thanos-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <[email protected]>

* Fix some linter warnings

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless logs

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <[email protected]>

* Remove TODO comment

Signed-off-by: Douglas Camata <[email protected]>

* Fix query tests

Signed-off-by: Douglas Camata <[email protected]>

* Reformat query docs

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless return

Signed-off-by: Douglas Camata <[email protected]>

* Put back old query docs

Signed-off-by: Douglas Camata <[email protected]>

* Update query docs again

Signed-off-by: Douglas Camata <[email protected]>

* Fix e2e env name

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <[email protected]>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <[email protected]>

* Clean up test

Signed-off-by: Douglas Camata <[email protected]>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <[email protected]>

* Remove extra empty line

Signed-off-by: Douglas Camata <[email protected]>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <[email protected]>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <[email protected]>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: utukj <[email protected]>

* docs: mark me as shepherd for next release (thanos-io#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: utukj <[email protected]>

* Revert "docs: mark me as shepherd for next release (thanos-io#5797)"

This reverts commit ea646a6.

Signed-off-by: utukj <[email protected]>

* Revert "Query: add query metrics to calls going through the Store API (thanos-io#5741)"

This reverts commit eec4fd0.

Signed-off-by: utukj <[email protected]>

* Revert "Receive: Reload tenant limit configuration on file change (thanos-io#5673)"

This reverts commit 24e1cc0.

Signed-off-by: utukj <[email protected]>

* Revert "Updates busybox SHA (thanos-io#5793)"

This reverts commit 9474c00.

Signed-off-by: utukj <[email protected]>

* Updates busybox SHA (thanos-io#5793)

Signed-off-by: GitHub <[email protected]>

Signed-off-by: GitHub <[email protected]>
Co-authored-by: yeya24 <[email protected]>
Signed-off-by: utukj <[email protected]>

* Receive: Reload tenant limit configuration on file change (thanos-io#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <[email protected]>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <[email protected]>

* Run goimports

Signed-off-by: Douglas Camata <[email protected]>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <[email protected]>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <[email protected]>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <[email protected]>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <[email protected]>

* Update lastReload

Signed-off-by: Douglas Camata <[email protected]>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <[email protected]>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <[email protected]>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <[email protected]>

* Run tests in parallel

Signed-off-by: Douglas Camata <[email protected]>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <[email protected]>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <[email protected]>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <[email protected]>

* Use absolute path to config file

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of parallel test

Signed-off-by: Douglas Camata <[email protected]>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless sleep

Signed-off-by: Douglas Camata <[email protected]>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <[email protected]>

* Remove unused fucntion

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <[email protected]>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <[email protected]>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <[email protected]>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <[email protected]>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <[email protected]>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <[email protected]>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: utukj <[email protected]>

* Query: add query metrics to calls going through the Store API (thanos-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <[email protected]>

* Fix some linter warnings

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless logs

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <[email protected]>

* Remove TODO comment

Signed-off-by: Douglas Camata <[email protected]>

* Fix query tests

Signed-off-by: Douglas Camata <[email protected]>

* Reformat query docs

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless return

Signed-off-by: Douglas Camata <[email protected]>

* Put back old query docs

Signed-off-by: Douglas Camata <[email protected]>

* Update query docs again

Signed-off-by: Douglas Camata <[email protected]>

* Fix e2e env name

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <[email protected]>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <[email protected]>

* Clean up test

Signed-off-by: Douglas Camata <[email protected]>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <[email protected]>

* Remove extra empty line

Signed-off-by: Douglas Camata <[email protected]>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <[email protected]>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <[email protected]>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: utukj <[email protected]>

* docs: mark me as shepherd for next release (thanos-io#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: utukj <[email protected]>

* Revert "docs: mark me as shepherd for next release (thanos-io#5797)"

This reverts commit c509c0e.

Signed-off-by: utukj <[email protected]>

* Revert "Updates busybox SHA (thanos-io#5793)"

This reverts commit ad11a03.

Signed-off-by: utukj <[email protected]>

* Revert "Query: add query metrics to calls going through the Store API (thanos-io#5741)"

This reverts commit 7a77769.

Signed-off-by: utukj <[email protected]>

* Revert "Receive: Reload tenant limit configuration on file change (thanos-io#5673)"

This reverts commit 32ca327.

Signed-off-by: utukj <[email protected]>

* fixed lint issue

Signed-off-by: utukj <[email protected]>

* added unit test for truncate and clean up

Signed-off-by: utukj <[email protected]>

* fixed truncate label func and added more tests

Signed-off-by: utukj <[email protected]>

* removed name from truncate test

Signed-off-by: utukj <[email protected]>

* reorganized test cases and removed redundant comments

Signed-off-by: utukj <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* fixed failing checks

Signed-off-by: utukj <[email protected]>

* e2e: Adding test for querier with two stores loadbalancing across them.

Signed-off-by: bwplotka <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* dumped long expected output in unittest

Signed-off-by: utukj <[email protected]>

* Revert "e2e: Adding test for querier with two stores loadbalancing across them."

This reverts commit 96b1545.

Signed-off-by: utukj <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* moved label definition to endpointset

Signed-off-by: utukj <[email protected]>

Signed-off-by: utukj <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>
Signed-off-by: GitHub <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: bwplotka <[email protected]>
Co-authored-by: Bartlomiej Plotka <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: yeya24 <[email protected]>
Co-authored-by: Douglas Camata <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Co-authored-by: Giedrius Statkevičius <[email protected]>
ngraham20 pushed a commit to ngraham20/thanos that referenced this pull request May 18, 2023
…specify metric labels to collect (thanos-io#5785)

* added unit tests for long labels and no external labels

Signed-off-by: utukj <[email protected]>

* trimmed too long external labels

Signed-off-by: utukj <[email protected]>

* added optional label selection

Signed-off-by: utukj <[email protected]>

* added cmd flag for choosing metric labels

Signed-off-by: utukj <[email protected]>

* updated docs

Signed-off-by: utukj <[email protected]>

* Update pkg/query/endpointset.go

full sentence fix

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* minor fixes from code review

Signed-off-by: utukj <[email protected]>

* fixed code comments

Signed-off-by: utukj <[email protected]>

* used enum for labels

Signed-off-by: utukj <[email protected]>

* updated query docs

Signed-off-by: utukj <[email protected]>

* cleaned up tests

Signed-off-by: utukj <[email protected]>

* Updates busybox SHA (thanos-io#5793)

Signed-off-by: GitHub <[email protected]>

Signed-off-by: GitHub <[email protected]>
Co-authored-by: yeya24 <[email protected]>
Signed-off-by: utukj <[email protected]>

* Receive: Reload tenant limit configuration on file change (thanos-io#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <[email protected]>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <[email protected]>

* Run goimports

Signed-off-by: Douglas Camata <[email protected]>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <[email protected]>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <[email protected]>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <[email protected]>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <[email protected]>

* Update lastReload

Signed-off-by: Douglas Camata <[email protected]>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <[email protected]>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <[email protected]>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <[email protected]>

* Run tests in parallel

Signed-off-by: Douglas Camata <[email protected]>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <[email protected]>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <[email protected]>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <[email protected]>

* Use absolute path to config file

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of parallel test

Signed-off-by: Douglas Camata <[email protected]>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless sleep

Signed-off-by: Douglas Camata <[email protected]>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <[email protected]>

* Remove unused fucntion

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <[email protected]>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <[email protected]>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <[email protected]>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <[email protected]>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <[email protected]>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <[email protected]>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: utukj <[email protected]>

* Query: add query metrics to calls going through the Store API (thanos-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <[email protected]>

* Fix some linter warnings

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless logs

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <[email protected]>

* Remove TODO comment

Signed-off-by: Douglas Camata <[email protected]>

* Fix query tests

Signed-off-by: Douglas Camata <[email protected]>

* Reformat query docs

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless return

Signed-off-by: Douglas Camata <[email protected]>

* Put back old query docs

Signed-off-by: Douglas Camata <[email protected]>

* Update query docs again

Signed-off-by: Douglas Camata <[email protected]>

* Fix e2e env name

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <[email protected]>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <[email protected]>

* Clean up test

Signed-off-by: Douglas Camata <[email protected]>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <[email protected]>

* Remove extra empty line

Signed-off-by: Douglas Camata <[email protected]>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <[email protected]>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <[email protected]>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: utukj <[email protected]>

* docs: mark me as shepherd for next release (thanos-io#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: utukj <[email protected]>

* Revert "docs: mark me as shepherd for next release (thanos-io#5797)"

This reverts commit ea646a6.

Signed-off-by: utukj <[email protected]>

* Revert "Query: add query metrics to calls going through the Store API (thanos-io#5741)"

This reverts commit eec4fd0.

Signed-off-by: utukj <[email protected]>

* Revert "Receive: Reload tenant limit configuration on file change (thanos-io#5673)"

This reverts commit 24e1cc0.

Signed-off-by: utukj <[email protected]>

* Revert "Updates busybox SHA (thanos-io#5793)"

This reverts commit 9474c00.

Signed-off-by: utukj <[email protected]>

* Updates busybox SHA (thanos-io#5793)

Signed-off-by: GitHub <[email protected]>

Signed-off-by: GitHub <[email protected]>
Co-authored-by: yeya24 <[email protected]>
Signed-off-by: utukj <[email protected]>

* Receive: Reload tenant limit configuration on file change (thanos-io#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <[email protected]>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <[email protected]>

* Run goimports

Signed-off-by: Douglas Camata <[email protected]>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <[email protected]>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <[email protected]>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <[email protected]>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <[email protected]>

* Update lastReload

Signed-off-by: Douglas Camata <[email protected]>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <[email protected]>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <[email protected]>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <[email protected]>

* Run tests in parallel

Signed-off-by: Douglas Camata <[email protected]>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <[email protected]>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <[email protected]>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <[email protected]>

* Use absolute path to config file

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of parallel test

Signed-off-by: Douglas Camata <[email protected]>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless sleep

Signed-off-by: Douglas Camata <[email protected]>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <[email protected]>

* Remove unused fucntion

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <[email protected]>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <[email protected]>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <[email protected]>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <[email protected]>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <[email protected]>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <[email protected]>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: utukj <[email protected]>

* Query: add query metrics to calls going through the Store API (thanos-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <[email protected]>

* Fix some linter warnings

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless logs

Signed-off-by: Douglas Camata <[email protected]>

* Refactor query tests

Signed-off-by: Douglas Camata <[email protected]>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <[email protected]>

* Remove TODO comment

Signed-off-by: Douglas Camata <[email protected]>

* Fix query tests

Signed-off-by: Douglas Camata <[email protected]>

* Reformat query docs

Signed-off-by: Douglas Camata <[email protected]>

* Remove useless return

Signed-off-by: Douglas Camata <[email protected]>

* Put back old query docs

Signed-off-by: Douglas Camata <[email protected]>

* Update query docs again

Signed-off-by: Douglas Camata <[email protected]>

* Fix e2e env name

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <[email protected]>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <[email protected]>

* Clean up test

Signed-off-by: Douglas Camata <[email protected]>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <[email protected]>

* Retrigger CI

Signed-off-by: Douglas Camata <[email protected]>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <[email protected]>

* Update changelog

Signed-off-by: Douglas Camata <[email protected]>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <[email protected]>

* Remove extra empty line

Signed-off-by: Douglas Camata <[email protected]>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <[email protected]>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <[email protected]>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <[email protected]>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <[email protected]>

Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Signed-off-by: utukj <[email protected]>

* docs: mark me as shepherd for next release (thanos-io#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: utukj <[email protected]>

* Revert "docs: mark me as shepherd for next release (thanos-io#5797)"

This reverts commit c509c0e.

Signed-off-by: utukj <[email protected]>

* Revert "Updates busybox SHA (thanos-io#5793)"

This reverts commit ad11a03.

Signed-off-by: utukj <[email protected]>

* Revert "Query: add query metrics to calls going through the Store API (thanos-io#5741)"

This reverts commit 7a77769.

Signed-off-by: utukj <[email protected]>

* Revert "Receive: Reload tenant limit configuration on file change (thanos-io#5673)"

This reverts commit 32ca327.

Signed-off-by: utukj <[email protected]>

* fixed lint issue

Signed-off-by: utukj <[email protected]>

* added unit test for truncate and clean up

Signed-off-by: utukj <[email protected]>

* fixed truncate label func and added more tests

Signed-off-by: utukj <[email protected]>

* removed name from truncate test

Signed-off-by: utukj <[email protected]>

* reorganized test cases and removed redundant comments

Signed-off-by: utukj <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* fixed failing checks

Signed-off-by: utukj <[email protected]>

* e2e: Adding test for querier with two stores loadbalancing across them.

Signed-off-by: bwplotka <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* dumped long expected output in unittest

Signed-off-by: utukj <[email protected]>

* Revert "e2e: Adding test for querier with two stores loadbalancing across them."

This reverts commit 96b1545.

Signed-off-by: utukj <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>

* moved label definition to endpointset

Signed-off-by: utukj <[email protected]>

Signed-off-by: utukj <[email protected]>
Signed-off-by: Uwakmfon Utuk <[email protected]>
Signed-off-by: GitHub <[email protected]>
Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Matej Gera <[email protected]>
Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: bwplotka <[email protected]>
Co-authored-by: Bartlomiej Plotka <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: yeya24 <[email protected]>
Co-authored-by: Douglas Camata <[email protected]>
Co-authored-by: Matej Gera <[email protected]>
Co-authored-by: Giedrius Statkevičius <[email protected]>

// Aggregate is an aggregator for merging `storepb.SeriesStatsCounter` for each incoming fanned out query.
func (s *seriesStatsAggregator) Aggregate(stats storepb.SeriesStatsCounter) {
s.seriesStats.Series += stats.Series

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't number of series overlap too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does, but we are (or were back then) not tracking the what individual series was touched. It would be quite expensive to do the tracking.

@@ -168,6 +169,7 @@ func (g *GRPCAPI) QueryRange(request *querypb.QueryRangeRequest, srv querypb.Que
request.EnableQueryPushdown,
false,
request.ShardInfo,
query.NoopSeriesStatsReporter,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this set to no-op here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because this is an internally initiated request, not something started by a user.

@christopherzli
Copy link

christopherzli commented May 16, 2024

just curious if we thought about expose these stats with the query response as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants