v1.2.0-rc.0

Breaking Changes

This release contains a number of small breaking changes. They will likely have no impact on your deployment, but it should be noted that due to a change in the API between the query-frontend and querier there may be a temporary read outage during deployment.

• [CHANGE] BREAKING CHANGE Drop support for v0 and v1 blocks. See 1.1 changelog for details #919 (@joe-elliott)
• [CHANGE] BREAKING CHANGE Consolidate status information onto /status endpoint #952 @zalegrala)
  The following endpoints moved.
  /runtime_config moved to /status/runtime_config
/config moved to /status/config
/services moved to /status/services
• [CHANGE] BREAKING CHANGE Change ingester metric ingester_bytes_metric_total in favor of ingester_bytes_received_total #979 (@mapno)
• [CHANGE] Renamed CLI flag from --storage.trace.maintenance-cycle to --storage.trace.blocklist_poll. This is a BREAKING CHANGE #897 (@mritunjaysharma394)
• [CHANGE] BREAKING CHANGE Support partial results from failed block queries #1007 (@mapno)
  Querier GET /querier/api/traces/<traceid> response's body has been modified
  to return tempopb.TraceByIDResponse instead of simply tempopb.Trace. This will cause a disruption of the read path during rollout of the change.
• [CHANGE] BRREAKING CHANGE Change the metrics name from cortex_runtime_config_last_reload_successful to tempo_runtime_config_last_reload_successful #945 (@kavirajk)

New Features and Enhancements

• [FEATURE] Add ability to search ingesters for traces #806 (@mdisibio @kvrhdn @annanay25)
• [FEATURE] Add runtime config handler #936 (@mapno)
• [FEATURE] Add ScalableSingleBinary operational run mode #1004 (@zalegrala)
• [ENHANCEMENT] Added "query blocks" cli option. #876 (@joe-elliott)
• [ENHANCEMENT] Added "search blocks" cli option. #972 (@joe-elliott)
• [ENHANCEMENT] Added traceid to trace too large message. #888 (@mritunjaysharma394)
• [ENHANCEMENT] Add support to tempo workloads to overrides from single configmap in microservice mode. #896 (@kavirajk)
• [ENHANCEMENT] Updated config defaults to reflect better capture operational knowledge. #913 (@joe-elliott)

  ingester:
    trace_idle_period: 30s => 10s  # reduce ingester memory requirements with little impact on querying
    flush_check_period: 30s => 10s
  query_frontend:
    query_shards: 2 => 20          # will massively improve performance on large installs
  storage:
    trace:
      wal:
        encoding: none => snappy   # snappy has been tested thoroughly and ready for production use
      block:
        bloom_filter_false_positive: .05 => .01          # will increase total bloom filter size but improve query performance
        bloom_filter_shard_size_bytes: 256KiB => 100 KiB # will improve query performance
  compactor:
    compaction:
      chunk_size_bytes: 10 MiB => 5 MiB  # will reduce compactor memory needs
      compaction_window: 4h => 1h        # will allow more compactors to participate in compaction without substantially increasing blocks
  

• [ENHANCEMENT] Make s3 backend readError logic more robust #905 (@wei840222)
• [ENHANCEMENT] Add gen index and gen bloom commands to tempo-cli. #903 (@annanay25)
• [ENHANCEMENT] Implement trace comparison in Vulture #904 (@zalegrala)
• [ENHANCEMENT] Compression updates: Added s2, improved snappy performance #961 (@joe-elliott)
• [ENHANCEMENT] Add support for vulture sending long running traces #951 (@zalegrala)
• [ENHANCEMENT] Shard tenant index creation by tenant and add functionality to handle stale indexes. #1005 (@joe-elliott)
• [ENHANCEMENT] Support partial results from failed block queries #1007 (@mapno)
• [ENHANCEMENT] Add new metric tempo_distributor_push_duration_seconds #1027 (@zalegrala)
• [ENHANCEMENT] Add query parameter to show the default config values and the difference between the current values and the defaults. #1045 (@MichelHollands)
• [ENHANCEMENT] Adding metrics around ingester flush retries #1049 (@dannykopping)
• [ENHANCEMENT] Performance: More efficient distributor batching #1075 (@joe-elliott)
• [ENHANCEMENT] Include tempo-cli in the release #1086 (@zalegrala)

Bug Fixes

• [BUGFIX] Update port spec for GCS docker-compose example #869 (@zalegrala)
• [BUGFIX] Fix "magic number" errors and other block mishandling when an ingester forcefully shuts down #937 (@mdisibio)
• [BUGFIX] Fix compactor memory leak #806 (@mdisibio)
• [BUGFIX] Set span's tag span.kind to client in query-frontend #975 (@mapno)
• [BUGFIX] Fixes tempodb_backend_hedged_roundtrips_total to correctly count hedged roundtrips. #1079 (@joe-elliott)

Other Changes

• [CHANGE] update jsonnet alerts and recording rules to use job_selectors and cluster_selectors for configurable unique identifier labels #935 (@kevinschoonover)
• [CHANGE] Add troubleshooting language to config for server.grpc_server_max_recv_msg_size and server.grpc_server_max_send_msg_size when handling large traces #1023 (@thejosephstevens)

v1.1.0

Breaking Changes

This release deprecates some internal data formats from prerelease versions of Tempo. If upgrading from Tempo v0.6.0 or earlier, then see the special upgrade instructions below. Tempo v0.7.0 and later have no compatibility issues or special instructions.

Tempo v0.6.0 and earlier used block formats v0 and v1, which are being deprecated, and support for these blocks will be removed in the next release. To resolve this you must first upgrade to Tempo 0.7.0+ (latest 1.1 is recommended) which introduces the supported v2 block format. Tempo will write all new blocks as v2, and it must continue running until all v0 and v1 blocks are gone (either deleted due to retention, or compacted). Block versions can be checked using the tempo-cli list blocks command.

New Features and Enhancements

[FEATURE] Added the ability to hedge requests with all backends #750 (@joe-elliott)
[FEATURE] Added a tenant index to reduce bucket polling. #834 (@joe-elliott)
[ENHANCEMENT] Added hedged request metric tempodb_backend_hedged_roundtrips_total and a new storage agnostic tempodb_backend_request_duration_seconds metric that supersedes the soon-to-be deprecated storage specific metrics (tempodb_azure_request_duration_seconds, tempodb_s3_request_duration_seconds and tempodb_gcs_request_duration_seconds). #790 (@josephwoodward)
[ENHANCEMENT] Performance: improve compaction speed with concurrent reads and writes #754 (@mdisibio)
[ENHANCEMENT] Improve readability of cpu and memory metrics on operational dashboard #764 (@bboreham)
[ENHANCEMENT] Add azure_request_duration_seconds metric. #767 (@josephwoodward)
[ENHANCEMENT] Add s3_request_duration_seconds metric. #776 (@josephwoodward)
[ENHANCEMENT] Add tempo_ingester_flush_size_bytes metric. #777 (@bboreham)
[ENHANCEMENT] Microservices jsonnet: resource requests and limits can be set in $._config. #793 (@kvrhdn)
[ENHANCEMENT] Add -config.expand-env cli flag to support environment variables expansion in config file. #796 (@Ashmita152)
[ENHANCEMENT] Add ability to control bloom filter caching based on age and/or compaction level. Add new cli command list cache-summary. #805 (@annanay25)
[ENHANCEMENT] Emit traces for ingester flush operations. #812 (@bboreham)
[ENHANCEMENT] Add retry middleware in query-frontend. #814 (@kvrhdn)
[ENHANCEMENT] Add -use-otel-tracer to use the OpenTelemetry tracer, this will also capture traces emitted by the gcs sdk. Experimental: not all features are supported (i.e. remote sampling). #842 (@kvrhdn)
[ENHANCEMENT] Add /services endpoint. #863 (@kvrhdn)
[ENHANCEMENT] Added "query blocks" cli option. #876 (@joe-elliott)
[ENHANCEMENT] Added traceid to trace too large message. #888 (@mritunjaysharma394)
[ENHANCEMENT] Reduce compactor memory usage by forcing garbage collection. #915 (@joe-elliott)

Bug Fixes

[BUGFIX] Allow only valid trace ID characters when decoding #854 (@zalegrala)
[BUGFIX] Queriers complete one polling cycle before finishing startup. #834 (@joe-elliott)
[BUGFIX] Update port spec for GCS docker-compose example #869 (@zalegrala)
[BUGFIX] Cortex upgrade to fix an issue where unhealthy compactors can't be forgotten #878 (@joe-elliott)

Other Changes

[CHANGE] Upgrade Cortex from v1.9.0 to v1.9.0-131-ga4bf10354 #841 (@aknuds1)
[CHANGE] Change example default tempo port from 3100 to 3200 #770 (@MurzNN)
[CHANGE] Jsonnet: use dedicated configmaps for distributors and ingesters #775 (@kvrhdn)
[CHANGE] Docker images are now prefixed by their branch name #828 (@jvrplmlmn)

v1.1.0-rc.1

Breaking Changes

This release deprecates some internal data formats from prerelease versions of Tempo. If upgrading from Tempo v0.6.0 or earlier, then see the special upgrade instructions below. Tempo v0.7.0 and later have no compatibility issues or special instructions.

Tempo v0.6.0 and earlier used block formats v0 and v1, which are being deprecated, and support for these blocks will be removed in the next release. To resolve this you must first upgrade to Tempo 0.7.0+ (latest 1.1 is recommended) which introduces the supported v2 block format. Tempo will write all new blocks as v2, and it must continue running until all v0 and v1 blocks are gone (either deleted due to retention, or compacted). Block versions can be checked using the tempo-cli list blocks command.

New Features and Enhancements

[FEATURE] Added the ability to hedge requests with all backends #750 (@joe-elliott)
[FEATURE] Added a tenant index to reduce bucket polling. #834 (@joe-elliott)
[ENHANCEMENT] Added hedged request metric tempodb_backend_hedged_roundtrips_total and a new storage agnostic tempodb_backend_request_duration_seconds metric that supersedes the soon-to-be deprecated storage specific metrics (tempodb_azure_request_duration_seconds, tempodb_s3_request_duration_seconds and tempodb_gcs_request_duration_seconds). #790 (@josephwoodward)
[ENHANCEMENT] Performance: improve compaction speed with concurrent reads and writes #754 (@mdisibio)
[ENHANCEMENT] Improve readability of cpu and memory metrics on operational dashboard #764 (@bboreham)
[ENHANCEMENT] Add azure_request_duration_seconds metric. #767 (@josephwoodward)
[ENHANCEMENT] Add s3_request_duration_seconds metric. #776 (@josephwoodward)
[ENHANCEMENT] Add tempo_ingester_flush_size_bytes metric. #777 (@bboreham)
[ENHANCEMENT] Microservices jsonnet: resource requests and limits can be set in $._config. #793 (@kvrhdn)
[ENHANCEMENT] Add -config.expand-env cli flag to support environment variables expansion in config file. #796 (@Ashmita152)
[ENHANCEMENT] Add ability to control bloom filter caching based on age and/or compaction level. Add new cli command list cache-summary. #805 (@annanay25)
[ENHANCEMENT] Emit traces for ingester flush operations. #812 (@bboreham)
[ENHANCEMENT] Add retry middleware in query-frontend. #814 (@kvrhdn)
[ENHANCEMENT] Add -use-otel-tracer to use the OpenTelemetry tracer, this will also capture traces emitted by the gcs sdk. Experimental: not all features are supported (i.e. remote sampling). #842 (@kvrhdn)
[ENHANCEMENT] Add /services endpoint. #863 (@kvrhdn)
[ENHANCEMENT] Added "query blocks" cli option. #876 (@joe-elliott)
[ENHANCEMENT] Added traceid to trace too large message. #888 (@mritunjaysharma394)

Bug Fixes

[BUGFIX] Allow only valid trace ID characters when decoding #854 (@zalegrala)
[BUGFIX] Queriers complete one polling cycle before finishing startup. #834 (@joe-elliott)
[BUGFIX] Update port spec for GCS docker-compose example #869 (@zalegrala)
[BUGFIX] Cortex upgrade to fix an issue where unhealthy compactors can't be forgotten #878 (@joe-elliott)

Other Changes

[CHANGE] Upgrade Cortex from v1.9.0 to v1.9.0-131-ga4bf10354 #841 (@aknuds1)
[CHANGE] Change example default tempo port from 3100 to 3200 #770 (@MurzNN)
[CHANGE] Jsonnet: use dedicated configmaps for distributors and ingesters #775 (@kvrhdn)
[CHANGE] Docker images are now prefixed by their branch name #828 (@jvrplmlmn)

v1.1.0-rc.0

Breaking Changes

• This release deprecates some internal data formats from prerelease versions of Tempo. If upgrading from Tempo v0.6.0 or earlier, then see the special upgrade instructions below. Tempo v0.7.0 and later have no compatibility issues or special instructions.

Tempo v0.6.0 and earlier used block formats v0 and v1, which are being deprecated, and support for these blocks will be removed in the next release. To resolve this you must first upgrade to Tempo 0.7.0+ (latest 1.1 is recommended) which introduces the supported v2 block format. Tempo will write all new blocks as v2, and it must continue running until all v0 and v1 blocks are gone (either deleted due to retention, or compacted). Block versions can be checked using the tempo-cli list blocks command.

New Features and Enhancements

[FEATURE] Added the ability to hedge requests with all backends #750 (@joe-elliott)
[FEATURE] Added a tenant index to reduce bucket polling. #834 (@joe-elliott)
[ENHANCEMENT] Added hedged request metric tempodb_backend_hedged_roundtrips_total and a new storage agnostic tempodb_backend_request_duration_seconds metric that supersedes the soon-to-be deprecated storage specific metrics (tempodb_azure_request_duration_seconds, tempodb_s3_request_duration_seconds and tempodb_gcs_request_duration_seconds). #790 (@josephwoodward)
[ENHANCEMENT] Performance: improve compaction speed with concurrent reads and writes #754 (@mdisibio)
[ENHANCEMENT] Improve readability of cpu and memory metrics on operational dashboard #764 (@bboreham)
[ENHANCEMENT] Add azure_request_duration_seconds metric. #767 (@josephwoodward)
[ENHANCEMENT] Add s3_request_duration_seconds metric. #776 (@josephwoodward)
[ENHANCEMENT] Add tempo_ingester_flush_size_bytes metric. #777 (@bboreham)
[ENHANCEMENT] Microservices jsonnet: resource requests and limits can be set in $._config. #793 (@kvrhdn)
[ENHANCEMENT] Add -config.expand-env cli flag to support environment variables expansion in config file. #796 (@Ashmita152)
[ENHANCEMENT] Add ability to control bloom filter caching based on age and/or compaction level. Add new cli command list cache-summary. #805 (@annanay25)
[ENHANCEMENT] Emit traces for ingester flush operations. #812 (@bboreham)
[ENHANCEMENT] Add retry middleware in query-frontend. #814 (@kvrhdn)
[ENHANCEMENT] Add -use-otel-tracer to use the OpenTelemetry tracer, this will also capture traces emitted by the gcs sdk. Experimental: not all features are supported (i.e. remote sampling). #842 (@kvrhdn)
[ENHANCEMENT] Add /services endpoint. #863 (@kvrhdn)

Bug Fixes

[BUGFIX] Allow only valid trace ID characters when decoding #854 (@zalegrala)
[BUGFIX] Queriers complete one polling cycle before finishing startup. #834 (@joe-elliott)

Other Changes

[CHANGE] Upgrade Cortex from v1.9.0 to v1.9.0-131-ga4bf10354 #841 (@aknuds1)
[CHANGE] Change example default tempo port from 3100 to 3200 #770 (@MurzNN)
[CHANGE] Jsonnet: use dedicated configmaps for distributors and ingesters #775 (@kvrhdn)
[CHANGE] Docker images are now prefixed by their branch name #828 (@jvrplmlmn)

v1.0.1

• [BUGFIX] Guard against negative dataLength #763 (@joe-elliott)

v1.0.0

Breaking changes

• This release contains a change to communication between distributors and ingesters which requires a specific rollout process to prevent dropped spans. First, rollout everything except distributors. After all ingesters have updated you can then rollout distributors to the latest version.
• -auth.enabled is marked deprecated. New flag is -multitenancy.enabled and is set to false by default. This is a breaking change if you were relying on auth/multitenancy being enabled by default. #646 @dgzlopes

Enhancements

This release contains significant improvements for performance and stability:

• [ENHANCEMENT] Performance: Improve Ingester Record Insertion. #681 @joe-elliott
• [ENHANCEMENT] Improve WAL Replay by not rebuilding the WAL. #668 @joe-elliott
• [ENHANCEMENT] Preallocate byte slices on ingester request unmarshal. #679 @annanay25
• [ENHANCEMENT] Reduce marshalling in the ingesters to improve performance. #694 @joe-elliott
• [ENHANCEMENT] Add config option to disable write extension to the ingesters. #677 @joe-elliott
• [ENHANCEMENT] Allow setting the bloom filter shard size with support dynamic shard count.#644 @annanay25
• [ENHANCEMENT] GCS SDK update v1.12.0 => v.15.0, ReadAllWithEstimate used in GCS/S3 backends. #693 @annanay25
• [ENHANCEMENT] Add a new endpoint /api/echo to test the query frontend is reachable. #714 @kvrhdn

Bugfixes

• [BUGFIX] Fix Query Frontend grpc settings to avoid noisy error log. #690 @annanay25
• [BUGFIX] Zipkin Support - CombineTraces. #688 @joe-elliott
• [BUGFIX] Zipkin support - Dedupe span IDs based on span.Kind (client/server) in Query Frontend. #687 @annanay25
• [BUGFIX] Azure Backend - Fix an issue with the append method on the Azure backend. #736 @pedrosaraiva

v1.0.0-rc.0

Breaking changes

• This release contains a change to communication between distributors and ingesters which requires a specific rollout process to prevent dropped spans. First, rollout everything except distributors. After all ingesters have updated you can then rollout distributors to the latest version.
• -auth.enabled is marked deprecated. New flag is -multitenancy.enabled and is set to false by default. This is a breaking change if you were relying on auth/multitenancy being enabled by default. #646 @dgzlopes

Enhancements

This release contains significant improvements for performance and stability:

• [ENHANCEMENT] Performance: Improve Ingester Record Insertion. #681 @joe-elliott
• [ENHANCEMENT] Improve WAL Replay by not rebuilding the WAL. #668 @joe-elliott
• [ENHANCEMENT] Preallocate byte slices on ingester request unmarshal. #679 @annanay25
• [ENHANCEMENT] Reduce marshalling in the ingesters to improve performance. #694 @joe-elliott
• [ENHANCEMENT] Add config option to disable write extension to the ingesters. #677 @joe-elliott
• [ENHANCEMENT] Allow setting the bloom filter shard size with support dynamic shard count.#644 @annanay25
• [ENHANCEMENT] GCS SDK update v1.12.0 => v.15.0, ReadAllWithEstimate used in GCS/S3 backends. #693 @annanay25
• [ENHANCEMENT] Add a new endpoint /api/echo to test the query frontend is reachable. #714 @kvrhdn

Bugfixes

• [BUGFIX] Fix Query Frontend grpc settings to avoid noisy error log. #690 @annanay25
• [BUGFIX] Zipkin Support - CombineTraces. #688 @joe-elliott
• [BUGFIX] Zipkin support - Dedupe span IDs based on span.Kind (client/server) in Query Frontend. #687 @annanay25

v0.7.0

License Change

• v0.7.0 and future versions are licensed under AGPLv3 #660

Breaking changes

• In an effort to move to byte based limits we removed some limits options and replaced them with byte based ones:
  max_spans_per_trace => max_bytes_per_trace
ingestion_rate_limit => ingestion_rate_limit_bytes
ingestion_burst_size => ingestion_burst_size_bytes
• The Query/QueryFrontend call signature has changed so there will be a query interruption during rollout.

All Changes

• [CHANGE] Update to Go 1.16, latest OpenTelemetry proto definition and collector #546 @mdisibio
• [CHANGE] max_spans_per_trace limit override has been removed in favour of max_bytes_per_trace.
  This is a breaking change to the overrides config section. #612 @annanay25
• [CHANGE] Add new flag -ingester.lifecycler.ID to manually override the ingester ID with which to register in the ring. #625 @annanay25
• [CHANGE] ingestion_rate_limit limit override has been removed in favour of ingestion_rate_limit_bytes.
  ingestion_burst_size limit override has been removed in favour of ingestion_burst_size_bytes.
  This is a breaking change to the overrides config section. #630 @annanay25

Features

• [FEATURE] Add page based access to the index file. #557 @joe-elliott
• [FEATURE] (Experimental) WAL Compression/checksums. #638 @joe-elliott

Enhancements

• [ENHANCEMENT] Add a Shutdown handler to flush data to backend, at "/shutdown". #526 @annanay25
• [ENHANCEMENT] Queriers now query all (healthy) ingesters for a trace to mitigate 404s on ingester rollouts/scaleups.
  This is a breaking change and will likely result in query errors on rollout as the query signature b/n QueryFrontend & Querier has changed. #557 @annanay25
• [ENHANCEMENT] Add list compaction-summary command to tempo-cli #588 @mdisibio
• [ENHANCEMENT] Add list and view index commands to tempo-cli #611 @mdisibio
• [ENHANCEMENT] Add a configurable prefix for HTTP endpoints. #631 @joe-elliott
• [ENHANCEMENT] Add kafka receiver. #613 @mapno
• [ENHANCEMENT] Upgrade OTel collector to v0.21.0. #613 @mapno
• [ENHANCEMENT] Add support for Cortex Background Cache. #640 @dgzlopes

Bugfixes

• [BUGFIX] Fixes permissions errors on startup in GCS. #554 @joe-elliott
• [BUGFIX] Fixes error where Dell ECS cannot list objects. #561 @kradalby
• [BUGFIX] Fixes listing blocks in S3 when the list is truncated. #567 @jojand
• [BUGFIX] Fixes where ingester may leave file open #570 @mdisibio
• [BUGFIX] Fixes a bug where some blocks were not searched due to query sharding and randomness in blocklist poll. #583 @annanay25
• [BUGFIX] Fixes issue where wal was deleted before successful flush and adds exponential backoff for flush errors #593 @mdisibio
• [BUGFIX] Fixes issue where Tempo would not parse odd length trace ids #605 @joe-elliott
• [BUGFIX] Sort traces on flush to reduce unexpected recombination work by compactors #606 @mdisibio
• [BUGFIX] Ingester fully persists blocks locally to reduce amount of work done after restart #628 @mdisibio

v0.6.0

0.6.0 release of Tempo comes with lots of features, enhancements and bugfixes. 🎁 🚀

The highlights of this release include support for Compression and Exhaustive search, along with CPU performance gains from changing compression algorithm between distributors and ingesters.

Breaking changes

[CHANGE] Ingester cut blocks based on size instead of trace count. Replace ingester traces_per_block setting with max_block_bytes. This is a breaking change. #474 (By default, Tempo will cut blocks of size 1GB) @mdisibio
[FEATURE] Added block compression. This is a breaking change b/c some configuration fields moved. #504 (By default, Tempo will use zstd for compression.) @joe-elliott
[CHANGE] Refactor cache section in tempodb. This is a breaking change b/c the cache config section has changed. #485 @dgzlopes

Enhancements

[CHANGE/BUGFIX] Rename tempodb_compaction_objects_written and tempodb_compaction_bytes_written metrics to tempodb_compaction_objects_written_total and tempodb_compaction_bytes_written_total. #524 @gouthamve
[ENHANCEMENT] Change default ingester_client compression from gzip to snappy. #522 @mdisibio
[ENHANCEMENT] Add exhaustive search to combine traces from all blocks in the backend. #489 @annanay25
[ENHANCEMENT] Add per-tenant block retention #77 @mdisibio
[ENHANCEMENT] Change index-downsample to index-downsample-bytes. This is a breaking change #519 @joe-elliott

Bugfixes

[BUGFIX] Upgrade cortex dependency to v1.7.0-rc.0+ to address issue with forgetting ring membership #442 #512 @mdisibio
[BUGFIX] No longer raise the tempodb_blocklist_poll_errors_total metric if a block doesn't have meta or compacted meta. #481 @joe-elliott
[BUGFIX] Replay wal completely before ingesting new spans. #525 @joe-elliott

For the full list of changes see CHANGELOG

v0.5.0

0.5.0 Release of Tempo with important new features and improvements. 🚀 Note: This release contains some breaking changes.

New Features

36991f0 Add support for Azure Blob Storage backend (#340) @pedrosaraiva
bc11b55 Add Query Frontend module to allow scaling the query path (#400) @annanay25

Breaking Changes

9e0e05a The gRPC signature from distributors to ingesters has changed. This is a breaking change when running in microservices mode with separate distributors and ingesters. To prevent errors first upgrade all ingesters, which adds the new gRPC endpoint, then upgrade the distributors. (#430) @mdisibio
dd7a18e Removed disk-based caching. Please migrate to redis or memcached. (#441) @joe-elliott
a796195 The ingestion_max_batch_size setting has been renamed to ingestion_burst_size. (#445) @mdisibio

Other changes and fixes

65a0643 Prevent race conditions between querier polling and ingesters clearing complete blocks (#421) @joe-elliott
4c0ea69 Exclude blocks in last active window from compaction (#411) @mdisibio
e804efd Mixin: Ignore metrics and query-frontend route when checking for TempoRequestLatency alert. (#440) @pstibrany
40abd5d Compactor without GCS permissions fail silently (#379) @mdisibio
40abd5d Add docker-compose example for GCS along with new backend options (#397) @mdisibio @achatterjee-grafana
00d9d3a Added tempo_distributor_bytes_received_total metric (#453) @joe-elliott
9e690dd Redo tempo-cli with basic command structure and improvements (#385) @mdisibio @achatterjee-grafana

For the full list of changes see CHANGELOG