Querying from grafana doesn't work with TLS

[lullen]

Describe the bug
Setting up tls on both http and grpc causes grafana to receive "Query error" when searching for spans.

I have followed the guide at https://grafana.com/docs/tempo/latest/configuration/tls/#client-configuration
It says to add querier.query-frontend.grpc_client_config however, that doesn't seem to exist but I found that there is a querier.frontend_worker.grpc_client_config which doesn't exist in the docs. Are they the same?

Everything else seems to work fine and I get messages these, so the ingestion seems to work fine.
level=info ts=2024-01-10T12:33:28.488848715Z caller=distributor.go:616 msg=received spanid=af253cd234c97243 traceid=b0a34725613194d4a3868db54c33492b

I do not get any errors at all in the logs in tempo but in grafana I see:
logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/uid/f64be1ec-e7c5-4824-addc-51f922c2477a/api/search remote_addr=::ffff:172.16.1.0 referer="https://grafana.app.local/explore?left=%7B%22datasource%22:%22f64be1ec-e7c5-4824-addc-51f922c2477a%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22tempo%22,%22uid%22:%22f64be1ec-e7c5-4824-addc-51f922c2477a%22%7D,%22queryType%22:%22nativeSearch%22,%22limit%22:20%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D&orgId=1" traceID=58b728656a8c1bc4a2d1d79b3928bb5d t=2024-01-10T12:53:30.495083622Z level=error msg="Proxy request failed" err="read tcp 172.16.1.72:56580->172.17.228.229:3200: read: connection reset by peer"

To Reproduce
Run grafana/tempo:2.3.1 container with the following config:

    multitenancy_enabled: false
    stream_over_http_enabled: true
    server:
      http_listen_port: 3200
      grpc_listen_port: 9095

      log_level: info

      grpc_tls_config:
        cert_file: /certs/tls.crt
        key_file: /certs/tls.key
      http_tls_config:
        cert_file: /certs/tls.crt
        key_file: /certs/tls.key
    query_frontend:
      search:
        duration_slo: 5s
        throughput_bytes_slo: 1.073741824e+09
      trace_by_id:
        duration_slo: 5s

    distributor:
      receivers:
        otlp:
          protocols:
            http:
              tls:
                cert_file: /certs/tls.crt
                key_file: /certs/tls.key
            grpc:
              tls:
                cert_file: /certs/tls.crt
                key_file: /certs/tls.key
      log_received_spans:
        enabled: true
    
    memberlist:
      tls_enabled: true
      tls_cert_path: /certs/tls.crt
      tls_key_path: /certs/tls.key

    ingester:
      max_block_duration: 5m               # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally
    ingester_client:
      grpc_client_config:
        tls_enabled: true
        tls_cert_path: /certs/tls.crt
        tls_key_path: /certs/tls.key
        tls_insecure_skip_verify: true

    compactor:
      compaction:
        block_retention: 720h                # overall Tempo trace retention

    metrics_generator:
      storage:
        path: /tmp/tempo/generator/wal
        remote_write:
          - url: https://prometheus-operated:9090/api/v1/write
            send_exemplars: true
            tls_config:
              insecure_skip_verify: true
    metrics_generator_client:
      grpc_client_config:
        tls_enabled: true
        tls_cert_path: /certs/tls.crt
        tls_key_path: /certs/tls.key
        tls_insecure_skip_verify: true

    storage:
      trace:
        backend: local                     # backend configuration to use
        wal:
          path: /tmp/tempo/wal             # where to store the the wal locally
        local:
          path: /tmp/tempo/blocks

    querier:
      frontend_worker:
        grpc_client_config:
          tls_enabled: true
          tls_cert_path: /certs/tls.crt
          tls_key_path: /certs/tls.key
          tls_insecure_skip_verify: true

    overrides:
      defaults:
        metrics_generator:
          processors: [service-graphs, span-metrics] # enables metrics generator

Expected behavior
Tempo should return the traces available and when not working, it should log the error

Environment:

• Infrastructure: Kubernetes

[joe-elliott]

I believe this is a known issue. The GRPC streaming endpoint is incompatible with TLS:

https://grafana.com/docs/tempo/latest/api_docs/#tempo-grpc-api

I'm working on a PR now to fix this that should be in 2.4.

[lullen]

Thank you.
Looking at the docs, it doesn't say anything about stream_over_http_enabled in the query_frontend settings. How do I enable it? In the beginning of the settings I do have stream_over_http_enabled: true

Edit: missed that its not compatible with tls

[ajschmidt8]

I believe this is a known issue. The GRPC streaming endpoint is incompatible with TLS:

https://grafana.com/docs/tempo/latest/api_docs/#tempo-grpc-api

I'm working on a PR now to fix this that should be in 2.4.

@joe-elliott, should this issue be fixed by now? I'm using a Tempo container from this recent commit and I'm still experiencing these TLS issues with Grafana 11.2.0.

[joe-elliott]

It should be fixed. This test starts up Tempo with TLS configured and exercises both the HTTP and GRPC query endpoints:

https://github.com/grafana/tempo/blob/main/integration/e2e/https_test.go#L27

[ajschmidt8]

It should be fixed. This test starts up Tempo with TLS configured and exercises both the HTTP and GRPC query endpoints:

https://github.com/grafana/tempo/blob/main/integration/e2e/https_test.go#L27

@joe-elliott, hmm. I'm still experiencing this issue when using Grafana + Tempo with mTLS (i.e. certs for both the Grafana client & Tempo server).

Does anything about that setup sound problematic to you? If I upload a docker-compose minimal reproducer, could you take a look?