Thanos Sidecar and Exemplars

[ioaiaaii]

Is your proposal related to a problem?

In our monitoring stack, we are using Thanos/Prometheus for metrics, Loki for Logs, and Grafana for visualization.
Now, we are in the progress of deploying Tempo for tracing and utilizing the exemplar feature, in order to have a unified solution for observability.

After upgrading Thanos to the latest version (currently v0.21.1), along with Prometheus operator and Grafana, we deployed Tempo. As a test case, we used tns demo app.

The topology is :

• Central Cluster: Grafana with Thanos as Datasource, Thanos(central with sidecar), Tempo
• Cluster App: Thanos (with sidecar), Grafana Agent, tns app

Test1
After tns deployment, we saw traces showing up in our Grafana using Tempo as a Data source.

Test2
After setting Cluster App's local Prometheus as a Data source, the exemplars appeared in tns Grafana dashboard.

Final Test
Switching Grafana's Data source to Thanos, since it's the main datastore we use for all our clusters, exemplars disappeared.

After searching in this repo, we noticed this issue and this PR as a follow-up solution.

In the documentation, there is no reference about exemplars in sidecar, and if an extra config has to be made.
In the latest documentation for the receiver, there is a reference about supporting exemplars, while in v0.21.1 is not.

So, this issue is more a question, of what we can do to support exemplars.
Currently, we set up Thanos with the sidecar model and we do not have the receiver.

Does the sidecar support exemplars, and we have to make configurations?
Is switching to the receive, the only way for exemplars?

Describe the solution you'd like

Additional context

We noticed that the option --tsdb.max-exemplars is not available in Thanos Reciever v0.21.1

[ioaiaaii]

It worked!

Leaving a comment here for reference.

In the centralized Thanos query, we parsed the following extra args :

extraFlags:
      - --exemplar=dnssrv+_grpc._tcp.prometheus-operator-XXXX.svc.cluster.local
      - --exemplar=thanos-querier.XXXCLUSTER1:80
      - --exemplar=thanos-querier.XXXCLUSTER2:80

Now, we have exemplars in our Dashboards, using our centralized Thanos as datasource.