Upgrade to thanos 0.32.1 #81
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
docs: mismatch in changelog
Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]>
…s-io#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <[email protected]> * Reverted test code Signed-off-by: Alex Le <[email protected]> * Updated comment Signed-off-by: Alex Le <[email protected]> * docs: mismatch in changelog Signed-off-by: Etienne Martel <[email protected]> Signed-off-by: Alex Le <[email protected]> * Updates busybox SHA (thanos-io#6283) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> Signed-off-by: Alex Le <[email protected]> * trigger workflow Signed-off-by: Alex Le <[email protected]> * trigger workflow Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> Signed-off-by: Etienne Martel <[email protected]> Signed-off-by: GitHub <[email protected]> Co-authored-by: Etienne Martel <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <[email protected]>
Signed-off-by: naveadkazi <[email protected]>
Add CarTrade Tech as new adopter
Signed-off-by: Walther Lee <[email protected]>
Signed-off-by: Walther Lee <[email protected]>
* Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <[email protected]> * Run go mod tidy Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]>
Signed-off-by: Walther Lee <[email protected]>
Signed-off-by: Walther Lee <[email protected]>
Signed-off-by: Walther Lee <[email protected]>
* Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <[email protected]> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <[email protected]> Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> Add chnagelog Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed changelog Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed qfe Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <[email protected]> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <[email protected]> Updated Readme Signed-off-by: Sebastian Rabenhorst <[email protected]> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <[email protected]> trigger tests Signed-off-by: Sebastian Rabenhorst <[email protected]> Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> * Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <[email protected]> * fixed imports Signed-off-by: Sebastian Rabenhorst <[email protected]> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <[email protected]> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <[email protected]> --------- Signed-off-by: Sebastian Rabenhorst <[email protected]>
* store: add streamed snappy encoding for postings list
We've noticed that decoding Snappy compressed postings list
takes a lot of RAM:
```
(pprof) top
Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total
Dropped 1069 nodes (cum <= 10.56GB)
Showing top 10 nodes out of 82
flat flat% sum% cum cum%
0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1
2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings
1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings
2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings
1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode
1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode
```
This is because we are creating a new []byte slice for the decoded data
each time. To avoid this RAM usage problem, let's stream the decoding
from a given buffer. Since Snappy block format doesn't support streamed
decoding, let's switch to Snappy stream format which is made for exactly
that.
Notice that our current `index.Postings` list does not
support going back through Seek() even if theoretically one could want
something like that. Fortunately, to search for posting intersection, we
need to only go forward.
Benchmark data:
```
name time/op
PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3%
PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4%
PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1%
PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4%
PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1%
PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13%
PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1%
PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4%
PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1%
PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4%
name alloc/op
PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0%
PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0%
name allocs/op
PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0%
PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0%
PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0%
PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0%
```
Compression ratios are still the same like previously:
```
$ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store
[snip]
=== RUN TestDiffVarintCodec/snappy/i!~"2.*"
postings_codec_test.go:73: postings entries: 944450
postings_codec_test.go:74: original size (4*entries): 3777800 bytes
postings_codec_test.go:80: encoded size 44498 bytes
postings_codec_test.go:81: ratio: 0.012
=== RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*"
postings_codec_test.go:73: postings entries: 944450
postings_codec_test.go:74: original size (4*entries): 3777800 bytes
postings_codec_test.go:80: encoded size 44670 bytes
postings_codec_test.go:81: ratio: 0.012
```
Signed-off-by: Giedrius Statkevičius <[email protected]>
* store: clean up postings code
Signed-off-by: Giedrius Statkevičius <[email protected]>
* store: fix estimation
Signed-off-by: Giedrius Statkevičius <[email protected]>
* store: use buffer.Bytes()
Signed-off-by: Giedrius Statkevičius <[email protected]>
* store/postings_codec: reuse extgrpc compressors/decompressors
Signed-off-by: Giedrius Statkevičius <[email protected]>
* CHANGELOG: add item
Signed-off-by: Giedrius Statkevičius <[email protected]>
* CHANGELOG: clean up whitespace
Signed-off-by: Giedrius Statkevičius <[email protected]>
---------
Signed-off-by: Giedrius Statkevičius <[email protected]>
With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Filip Petkovski <[email protected]>
Signed-off-by: Filip Petkovski <[email protected]>
Signed-off-by: Filip Petkovski <[email protected]>
…-heap-less-func Reuse buffers for label comparison
Signed-off-by: Walther Lee <[email protected]>
Signed-off-by: Sebastian Rabenhorst <[email protected]>
Signed-off-by: Philip Gough <[email protected]>
Signed-off-by: Walther Lee <[email protected]>
* Replaced summary in extprom metrics with histogram Signed-off-by: Sebastian Rabenhorst <[email protected]> * Added changelog Signed-off-by: Sebastian Rabenhorst <[email protected]> * Removed unused parameters from NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <[email protected]> * Reverted NewInstrumentationMiddleware Signed-off-by: Sebastian Rabenhorst <[email protected]> --------- Signed-off-by: Sebastian Rabenhorst <[email protected]>
This commit exposes the label set alongside the min and max time for each TSDB covered by a Store. This information is used to scope the min time for a remote query so that we do not produce partial aggregates in distriuted mode. Signed-off-by: Filip Petkovski <[email protected]>
Signed-off-by: Filip Petkovski <[email protected]>
Signed-off-by: Sebastian Rabenhorst <[email protected]> Cleanup Signed-off-by: Sebastian Rabenhorst <[email protected]>
Signed-off-by: Sebastian Rabenhorst <[email protected]>
Pass QueryEngineFactory by reference
…o#6322) Signed-off-by: Xiaochao Dong (@damnever) <[email protected]>
…-heap-less-func Ignore common label-values in ProxyResponseHeap sort function
Signed-off-by: Saswata Mukherjee <[email protected]>
Chunk reader needs to wait until the chunk loading ends in Close() because otherwise there will be a race between appending to r.chunkBytes and reading from it. Signed-off-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Saswata Mukherjee <[email protected]>
Signed-off-by: Saswata Mukherjee <[email protected]>
* store: fix missing flush when handling pushed down queries (thanos-io#6612) In the case that we have pushed down queries and internal labels that are overriden by external labels we are not flushing the sorted response. Signed-off-by: Michael Hoffmann <[email protected]> * Cut release candidate v0.32.0-rc.1 Signed-off-by: Saswata Mukherjee <[email protected]> --------- Signed-off-by: Michael Hoffmann <[email protected]> Signed-off-by: Saswata Mukherjee <[email protected]> Co-authored-by: Michael Hoffmann <[email protected]>
* queryfrontend: fix explanation with query_range (thanos-io#6633) Signed-off-by: Saswata Mukherjee <[email protected]> * Cut final release candidate for v0.32.0 Signed-off-by: Saswata Mukherjee <[email protected]> --------- Signed-off-by: Saswata Mukherjee <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]>
Signed-off-by: Subbarao Meduri <[email protected]>
subbarao-meduri
requested review from
douglascamata,
saswatamcode and
philipgough
|
/retest |
Signed-off-by: Subbarao Meduri <[email protected]>
Signed-off-by: Subbarao Meduri <[email protected]>
Signed-off-by: Subbarao Meduri <[email protected]>
* store: fix error handling in decodePostings (thanos-io#6650) Signed-off-by: Michael Hoffmann <[email protected]> * store: fix ignored error in postings (thanos-io#6654) Signed-off-by: Michael Hoffmann <[email protected]> * Store: fix bufio pool handling (thanos-io#6655) Signed-off-by: Michael Hoffmann <[email protected]> * Fix mutable stringset memory usage (thanos-io#6669) This commit fixes the Insert function for the mutable stringset to only insert unique labels instead of adding every label to the set. Signed-off-by: Filip Petkovski <[email protected]> * Cut patch release v0.32.1 Signed-off-by: Saswata Mukherjee <[email protected]> --------- Signed-off-by: Michael Hoffmann <[email protected]> Signed-off-by: Filip Petkovski <[email protected]> Signed-off-by: Saswata Mukherjee <[email protected]> Co-authored-by: Michael Hoffmann <[email protected]> Co-authored-by: Filip Petkovski <[email protected]>
v0.32.1 Signed-off-by: Subbarao Meduri <[email protected]>
Signed-off-by: Subbarao Meduri <[email protected]>
Signed-off-by: Subbarao Meduri <[email protected]>
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bjoydeep, subbarao-meduri The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
TODO: Replace releaese-2.9 branch with contents o this branch
Changes:
kindtesting of the upstream image in prow buildsVerification