Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge upstream f6871f7 (thanos-io#149)
* mixins: Add code/grpc-code dimension to error widgets Signed-off-by: Douglas Camata <[email protected]> * Update changelog Signed-off-by: Douglas Camata <[email protected]> * Fix messed up merge conflict resolution Signed-off-by: Douglas Camata <[email protected]> * Readd empty line at the end of changelog Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> * mixin(Rule): Add rule evaluation failures to the Rule dashboard (thanos-io#6244) * Improve Thanos Rule dashboard legends Signed-off-by: Douglas Camata <[email protected]> * Add evaluations failed to Rule dashboard Signed-off-by: Douglas Camata <[email protected]> * Refactor rule dashboard Signed-off-by: Douglas Camata <[email protected]> * Add changelog entry Signed-off-by: Douglas Camata <[email protected]> * Rerun CI Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> * added thanos logo in react app (thanos-io#6264) Signed-off-by: hackeramitkumar <[email protected]> * Add an experimental flag to block samples with timestamp too far in the future (thanos-io#6195) * Add an experimental flag to block samples with timestamp too far in the future Signed-off-by: Yi Jin <[email protected]> * fix bug Signed-off-by: Yi Jin <[email protected]> * address comments Signed-off-by: Yi Jin <[email protected]> * fix docs CI errors Signed-off-by: Yi Jin <[email protected]> * resolve merge conflicts Signed-off-by: Yi Jin <[email protected]> * resolve merge conflicts Signed-off-by: Yi Jin <[email protected]> * retrigger checks Signed-off-by: Yi Jin <[email protected]> --------- Signed-off-by: Yi Jin <[email protected]> * store/bucket: snappy-encoded postings reading improvements (thanos-io#6245) * store: pool input to snappy.Decode Pool input to snappy.Decode to avoid allocations. Signed-off-by: Giedrius Statkevičius <[email protected]> * store: use s2 for decoding snappy It's faster hence use it. Signed-off-by: Giedrius Statkevičius <[email protected]> * store: small code style adjustment Signed-off-by: Giedrius Statkevičius <[email protected]> * store: call closefns before returning err Signed-off-by: Giedrius Statkevičius <[email protected]> * store/postings_codec: return both if possible Signed-off-by: Giedrius Statkevičius <[email protected]> * store/bucket: always call close fns Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * truncateExtLabels support Unicode cut (thanos-io#6267) * truncateExtLabels support Unicode cut Signed-off-by: mickeyzzc <[email protected]> * update TestTruncateExtLabels and pass test Signed-off-by: mickeyzzc <[email protected]> --------- Signed-off-by: mickeyzzc <[email protected]> * Update mentorship links Signed-off-by: Saswata Mukherjee <[email protected]> * Fix segfault in LabelValues during head compaction (thanos-io#6271) * Fix segfault in LabelValues during head compaction Head compaction causes blocks outside the retention period to get deleted. If there is an in-flight LabelValues request at the same time, deleting the block can cause the store proxy to panic since it loses access to the data. This commit fixes the issue by copying label values from TSDB stores before returning them to the store proxy. I thought about exposing a Close method on the TSDB store which the Proxy can call, but this will not eliminate cases where gRPC defers sending data over a channel using its queueing mechanism. Signed-off-by: Filip Petkovski <[email protected]> * Add changelog entry Signed-off-by: Filip Petkovski <[email protected]> * Assert no error when querying labels Signed-off-by: Filip Petkovski <[email protected]> --------- Signed-off-by: Filip Petkovski <[email protected]> * Mixin: Allow specifying an instance name filter (thanos-io#6273) This commit allow specifying the instance name filter, in order to filter the datasources shown on the dashboards. For example, when generating the dashboards one can do the following (i.e in config.libsonnet) ``` dashboard+:: { prefix: 'Thanos / ', ... instance_name_filter: '/EU.*/' ``` Signed-off-by: Jacob Baungard Hansen <[email protected]> * Adds Deno to adopters.yml (thanos-io#6275) Signed-off-by: Will (Newby) Atlas <[email protected]> * Bump `make test` timeout (thanos-io#6276) Signed-off-by: Douglas Camata <[email protected]> * fix 0.31 changelog (thanos-io#6278) Signed-off-by: junot <[email protected]> * Query: Switch Multiple Engines (thanos-io#6234) * Query: Switch engines using `engine` param Thanos query has two engine, prometheus (default) and thanos. A single engine runs through thanos query command at a time, and have to re run the command to switch between. This commit adds a functionality to run multiple engines at once and switch between them using `engine` query param inq query api. To avoid duplicate matrics registration, the thanos engine is provided with a different registerer having prefix `tpe_` (not been finalized yet). promql-engine command line flag has been removed that specifies the query engine to run. Currently this functionality not implemented on GRPCAPI. Signed-off-by: Pradyumna Krishna <[email protected]> * Add multiple engine support to GRPCAPI Fix build fail for thanos, adds support for multiple engine in GRPCAPI. Signed-off-by: Pradyumna Krishna <[email protected]> * Create QueryEngineFactory to create engines QueryEngineFactory makes a collection for all promql engines used by thanos and returns it. Any engine can be created and returned using `GetXEngine` method. It is currently limited to 2 engines prometheus and thanos engines that get created on the first call. Signed-off-by: Pradyumna Krishna <[email protected]> * Use QueryEngineFactory in query API thanos query commands pass `QueryEngineFactory` to query apis that will use engine based on query params. It will provide more flexibility to create multiple engines in thanos. Adds `defaultEngine` CLI flag, A default engine to use if not specified with query params. Signed-off-by: Pradyumna Krishna <[email protected]> * Update Query API tests Fixes breaking tests Signed-off-by: Pradyumna Krishna <[email protected]> * Minor changes and Docs fixes * Move defaultEngine argument to reduce diff. * Generated Docs. Signed-off-by: Pradyumna Krishna <[email protected]> * Add Engine Selector/ Dropdown to Query UI Engine Selector is a dropdown that sets an engine to be used to run the query. Currently two engines `thanos` and `prometheus`. This dropdown sends a query param `engine` to query api, which runs the api using the engine provided. Provided to run query using multiple query engines from Query UI. Signed-off-by: Pradyumna Krishna <[email protected]> * Move Engine Selector to Panel Removes Dropdown component, and renders Engine Selector directly. Receives defaultEngine from `flags` API. Updates parseOptions to parse engine query param and updates test for Panel and utils. Signed-off-by: Pradyumna Krishna <[email protected]> * Upgrade promql-engine dependency Updates promql-engine that brings functionality to provide fallback engine using enigne Opts. Signed-off-by: Pradyumna Krishna <[email protected]> * Add MinT to remote client MinT method was missing from Client due to updated promql-engine. This commits adds mint to the remote client. Signed-off-by: Pradyumna Krishna <[email protected]> * Use prometheus fallback engine in thanos engine Thanos engine creates a fallback prometheus engine that conflicts with another prometheus engine created by thanos, while registering metrics. To fix this, provided created thanos engine as fallback engine to thanos engine in engine Opts. Signed-off-by: Pradyumna Krishna <[email protected]> * Use enum for EngineType in GRPC GRPC is used for communication between thanos components and defaultEngine was a string before. Enum makes more sense, and hence the request.Enigne type has been changed to querypb.EngineType. Default case is handled with another default value provided in the enum. Signed-off-by: Pradyumna Krishna <[email protected]> * Update query UI bindata.go Compile react app using `make assets`. Signed-off-by: Pradyumna Krishna <[email protected]> --------- Signed-off-by: Pradyumna Krishna <[email protected]> * docs: mismatch in changelog Signed-off-by: Etienne Martel <[email protected]> * Updates busybox SHA (thanos-io#6283) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 (thanos-io#6287) * Upgrade prometheus to 7309ac272195cb856b879306d6a27af7641d3346 Signed-off-by: Alex Le <[email protected]> * Reverted test code Signed-off-by: Alex Le <[email protected]> * Updated comment Signed-off-by: Alex Le <[email protected]> * docs: mismatch in changelog Signed-off-by: Etienne Martel <[email protected]> Signed-off-by: Alex Le <[email protected]> * Updates busybox SHA (thanos-io#6283) Signed-off-by: GitHub <[email protected]> Co-authored-by: fpetkovski <[email protected]> Signed-off-by: Alex Le <[email protected]> * trigger workflow Signed-off-by: Alex Le <[email protected]> * trigger workflow Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> Signed-off-by: Etienne Martel <[email protected]> Signed-off-by: GitHub <[email protected]> Co-authored-by: Etienne Martel <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <[email protected]> * Add CarTrade Tech as new adopter Signed-off-by: naveadkazi <[email protected]> * tests: Remove custom Between test matcher (thanos-io#6310) * Remove custom Between test matcher The upstream PR to efficientgo/e2e has been merged, so we can use it from there. Signed-off-by: Douglas Camata <[email protected]> * Run go mod tidy Signed-off-by: Douglas Camata <[email protected]> --------- Signed-off-by: Douglas Camata <[email protected]> * query frontend, query UI: Native histogram support (thanos-io#6071) * Implemented native histogram support for qfe and query UI Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed marshalling for histograms in qfe Started working on native histogram query ui Copied histogram implementation for graph Added query range support for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Use prom model (un-)marshal for native histograms in qfe Fixed sample and sample stream marshal fn Extended qfe native histogram e2e tests Added copyright to qfe queryrange compat Added query range test fo histograms and try to fix ui tests Fixed DataTable test Review feedback Fixed native histogram e2e test Signed-off-by: Sebastian Rabenhorst <[email protected]> Add histogram support for ApplyCounterResetsSeriesIterator Signed-off-by: Sebastian Rabenhorst <[email protected]> Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> Add chnagelog Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed changelog Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed qfe Signed-off-by: Sebastian Rabenhorst <[email protected]> Fixed PrometheusResponse minTime for histograms in qfe Signed-off-by: Sebastian Rabenhorst <[email protected]> Updated prometheus common to v0.40.0 and queryrange.Sample fixes Signed-off-by: Sebastian Rabenhorst <[email protected]> Updated Readme Signed-off-by: Sebastian Rabenhorst <[email protected]> Addressed PR comments Signed-off-by: Sebastian Rabenhorst <[email protected]> trigger tests Signed-off-by: Sebastian Rabenhorst <[email protected]> Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> * Made assets Signed-off-by: Sebastian Rabenhorst <[email protected]> * fixed tsdbutil references Signed-off-by: Sebastian Rabenhorst <[email protected]> * fixed imports Signed-off-by: Sebastian Rabenhorst <[email protected]> * Enabled pushdown for query native hist test and removed ToDo Signed-off-by: Sebastian Rabenhorst <[email protected]> * Refactored native histogram query UI Signed-off-by: Sebastian Rabenhorst <[email protected]> --------- Signed-off-by: Sebastian Rabenhorst <[email protected]> * store: add streamed snappy encoding for postings list (thanos-io#6303) * store: add streamed snappy encoding for postings list We've noticed that decoding Snappy compressed postings list takes a lot of RAM: ``` (pprof) top Showing nodes accounting for 1427.30GB, 67.55% of 2112.82GB total Dropped 1069 nodes (cum <= 10.56GB) Showing top 10 nodes out of 82 flat flat% sum% cum cum% 0 0% 0% 1905.67GB 90.20% golang.org/x/sync/errgroup.(*Group).Go.func1 2.08GB 0.098% 0.098% 1456.94GB 68.96% github.com/thanos-io/thanos/pkg/store.(*blockSeriesClient).ExpandPostings 1.64GB 0.078% 0.18% 1454.87GB 68.86% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).ExpandedPostings 2.31GB 0.11% 0.29% 1258.15GB 59.55% github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).fetchPostings 1.48GB 0.07% 0.36% 1219.67GB 57.73% github.com/thanos-io/thanos/pkg/store.diffVarintSnappyDecode 1215.21GB 57.52% 57.87% 1215.21GB 57.52% github.com/klauspost/compress/s2.Decode ``` This is because we are creating a new []byte slice for the decoded data each time. To avoid this RAM usage problem, let's stream the decoding from a given buffer. Since Snappy block format doesn't support streamed decoding, let's switch to Snappy stream format which is made for exactly that. Notice that our current `index.Postings` list does not support going back through Seek() even if theoretically one could want something like that. Fortunately, to search for posting intersection, we need to only go forward. Benchmark data: ``` name time/op PostingsEncodingDecoding/10000/raw/encode-16 71.6µs ± 3% PostingsEncodingDecoding/10000/raw/decode-16 76.3ns ± 4% PostingsEncodingDecoding/10000#01/snappy/encode-16 73.3µs ± 1% PostingsEncodingDecoding/10000#01/snappy/decode-16 1.63µs ± 6% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 111µs ± 2% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 14.5µs ± 7% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 1.09ms ± 2% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 14.4µs ± 4% PostingsEncodingDecoding/100000#01/raw/encode-16 710µs ± 1% PostingsEncodingDecoding/100000#01/raw/decode-16 79.3ns ±13% PostingsEncodingDecoding/100000#02/snappy/encode-16 719µs ± 1% PostingsEncodingDecoding/100000#02/snappy/decode-16 13.5µs ± 4% PostingsEncodingDecoding/1000000/raw/encode-16 7.14ms ± 1% PostingsEncodingDecoding/1000000/raw/decode-16 81.7ns ± 9% PostingsEncodingDecoding/1000000#01/snappy/encode-16 7.52ms ± 3% PostingsEncodingDecoding/1000000#01/snappy/decode-16 139µs ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 11.4ms ± 4% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 15.5µs ± 4% name alloc/op PostingsEncodingDecoding/10000/raw/encode-16 13.6kB ± 0% PostingsEncodingDecoding/10000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 25.9kB ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 11.0kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 16.6kB ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 148kB ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 148kB ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 131kB ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 254kB ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 107kB ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 1.25MB ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 96.0B ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 2.48MB ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 1.05MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 1.47MB ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 148kB ± 0% name allocs/op PostingsEncodingDecoding/10000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/10000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/10000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/10000#02/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/100000/snappyStreamed/decode-16 5.00 ± 0% PostingsEncodingDecoding/100000#01/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/100000#01/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/100000#02/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000/raw/encode-16 2.00 ± 0% PostingsEncodingDecoding/1000000/raw/decode-16 2.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/encode-16 3.00 ± 0% PostingsEncodingDecoding/1000000#01/snappy/decode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/encode-16 4.00 ± 0% PostingsEncodingDecoding/1000000#02/snappyStreamed/decode-16 5.00 ± 0% ``` Compression ratios are still the same like previously: ``` $ /bin/go test -v -timeout 10m -run ^TestDiffVarintCodec$ github.com/thanos-io/thanos/pkg/store [snip] === RUN TestDiffVarintCodec/snappy/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44498 bytes postings_codec_test.go:81: ratio: 0.012 === RUN TestDiffVarintCodec/snappyStreamed/i!~"2.*" postings_codec_test.go:73: postings entries: 944450 postings_codec_test.go:74: original size (4*entries): 3777800 bytes postings_codec_test.go:80: encoded size 44670 bytes postings_codec_test.go:81: ratio: 0.012 ``` Signed-off-by: Giedrius Statkevičius <[email protected]> * store: clean up postings code Signed-off-by: Giedrius Statkevičius <[email protected]> * store: fix estimation Signed-off-by: Giedrius Statkevičius <[email protected]> * store: use buffer.Bytes() Signed-off-by: Giedrius Statkevičius <[email protected]> * store/postings_codec: reuse extgrpc compressors/decompressors Signed-off-by: Giedrius Statkevičius <[email protected]> * CHANGELOG: add item Signed-off-by: Giedrius Statkevičius <[email protected]> * CHANGELOG: clean up whitespace Signed-off-by: Giedrius Statkevičius <[email protected]> --------- Signed-off-by: Giedrius Statkevičius <[email protected]> * compact: atomically replace no compact marked map (thanos-io#6319) With lots of blocks it could take some time to fill this no compact marked map hence replace it atomically. I believe this leads to problems in the compaction planner where it picks up no compact marked blocks because meta syncer does synchronizations concurrently. Signed-off-by: Giedrius Statkevičius <[email protected]> * Fixed modules, logicalplan flag and more * Made assets * Removed unused test function * Sorted labels --------- Signed-off-by: Douglas Camata <[email protected]> Signed-off-by: hackeramitkumar <[email protected]> Signed-off-by: Yi Jin <[email protected]> Signed-off-by: Giedrius Statkevičius <[email protected]> Signed-off-by: mickeyzzc <[email protected]> Signed-off-by: Saswata Mukherjee <[email protected]> Signed-off-by: Filip Petkovski <[email protected]> Signed-off-by: Jacob Baungard Hansen <[email protected]> Signed-off-by: Will (Newby) Atlas <[email protected]> Signed-off-by: junot <[email protected]> Signed-off-by: Pradyumna Krishna <[email protected]> Signed-off-by: Etienne Martel <[email protected]> Signed-off-by: GitHub <[email protected]> Signed-off-by: Alex Le <[email protected]> Signed-off-by: naveadkazi <[email protected]> Signed-off-by: Sebastian Rabenhorst <[email protected]> Co-authored-by: Douglas Camata <[email protected]> Co-authored-by: Filip Petkovski <[email protected]> Co-authored-by: Amit kumar <[email protected]> Co-authored-by: Yi Jin <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]> Co-authored-by: MickeyZZC <[email protected]> Co-authored-by: Saswata Mukherjee <[email protected]> Co-authored-by: Jacob Baungård Hansen <[email protected]> Co-authored-by: Will (Newby) Atlas <[email protected]> Co-authored-by: junot <[email protected]> Co-authored-by: Pradyumna Krishna <[email protected]> Co-authored-by: Etienne Martel <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: fpetkovski <[email protected]> Co-authored-by: Alex Le <[email protected]> Co-authored-by: naveadkazi <[email protected]>
- Loading branch information
17 people
authored
May 2, 2023
1 parent
a45abcb
commit 8f1855a