v2.4.0
Breaking Changes
- Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics - Deprecating vParquet v1 #3377 (@ie-pham)
vParquet
can no longer be explicitly configured as a block version. Tempo 2.4 will continue to read these blocks. Support will be dropped entirely in 2.5. The current/default block version isvParquet3
. - Major cache refactor to allow multiple role based caches to be configured #3166 (@joe-elliott).
Deprecate the following fields. These have all been migrated to a top level "cache:" field.storage: trace: cache: search: cache_control: background_cache: memcached: redis:
Changes
- [CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
- [CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
- [CHANGE] Set
autocomplete_filtering_enabled
totrue
by default #3178 (@mapno) - [CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
- [CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham)
Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors
The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics - [CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
- [CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)
Features
- [FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
- [FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
- [FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166.
BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.storage: trace: cache: search: cache_control: background_cache: memcached: redis:
Enhancements
- [ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
- [ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
- [ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
- [ENHANCEMENT] Introduced
AttributePolicyMatch
&IntrinsicPolicyMatch
structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk) - [ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
- [ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
- [ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
- [ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
- [ENHANCEMENT] Added a
frontend-search
cache role for job search caching. #3225 (@joe-elliott) - [ENHANCEMENT] Added a
parquet-page
cache role for page level caching. #3196 (@joe-elliott) - [ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
- [ENHANCEMENT] Add
--max-start-time
and--min-start-time
flag to tempo-cli commandanalyse blocks
#3250 (@mapno) - [ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
- [ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
- [ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
- [ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
- [ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
- [ENHANCEMENT] Add
--shutdown-delay
to allow Tempo to cleanly drain connections. #3395 (@joe-elliott) - [ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303#3303 (@zalegrala)
- [ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
- [ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
- [ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
- [ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)
- [ENHANCEMENT] Config: Adds
query-frontend.log-query-request-headers
to enable logging of request headers in query logs. #3383 (@jmichalek132)
Bugfixes
- [BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
- [BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
- [BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
- [BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
- [BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
- [BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
- [BUGFIX] Restore
tempo_request_duration_seconds
metrics forquerier_api_*
requests #3403 (@kvrhdn) - [BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
- [BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
- [BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
- [BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
- [BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
- [BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
- [BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
- [BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)
- [BUGFIX] Fix some instances where spanmetrics histograms could be inconsistent #3412 (@mdisibio)