Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

prometheusexporter not attaching exemplars to metrics generated from spanmetricsconnector #28852

Closed
ctrader2 opened this issue Nov 1, 2023 · 4 comments
Closed

prometheusexporter not attaching exemplars to metrics generated from spanmetricsconnector #28852

ctrader2 opened this issue Nov 1, 2023 · 4 comments
Labels
bug Something isn't working connector/spanmetrics exporter/prometheus needs triage New item requiring triage waiting for author

Comments

@ctrader2
Copy link

ctrader2 commented Nov 1, 2023

Component(s)

connector/spanmetrics, exporter/prometheus

What happened?

Description

I am using the spanmetricsconnector in conjunction with prometheusexporter. I want to attach exemplars to the Prometheus metrics being generated. Based on the documentation for each of these components, I configured exemplars:enabled:true for the spanmetricsconnector and enable_open_metrics:true for the prometheusexporter.

It looks like exemplars are attached correctly to the OTEL metrics generated by spanmetricsconnector but are not carried forward/attached after conversion to Prometheus format.

Steps to Reproduce

  1. Run the collector with the specified configuration
  2. Send traces to the collector
  3. View the Prometheus metrics which are generated from the trace data using the spanmetricsconnector

Expected Result

Trace IDs attached to each Prometheus metric.

Actual Result

No trace information is attached.

Collector version

v0.88.0

Environment information

No response

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:

processors:
  batch:
  filter/httponly:
    error_mode: ignore
    metrics:
      datapoint:
        - 'attributes["span.kind"] != "SPAN_KIND_SERVER" or attributes["http.status_code"] == nil' 

exporters:
  otlp:
    endpoint: grafana-agent:4317
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
    enable_open_metrics: true
    add_metric_suffixes: false
  logging:
    verbosity: detailed

connectors:
  spanmetrics:
    dimensions:
      - name: http.method
      - name: http.status_code
      - name: http.route
    exemplars:
      enabled: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [spanmetrics, otlp]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      processors: [filter/httponly]
      exporters: [logging, prometheus]

Log output

Collector logs:

jobrunner-otel-collector_1      | 2023-11-01T19:16:29.040Z      info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 2, "data points": 2}
jobrunner-otel-collector_1      | 2023-11-01T19:16:29.040Z      info    ResourceMetrics #0
jobrunner-otel-collector_1      | Resource SchemaURL:
jobrunner-otel-collector_1      | Resource attributes:
jobrunner-otel-collector_1      |      -> container.id: Str(3973966180a2a8394ff4d60c3e2806e7486725dae9b2595a64bc6b7db59effd9)
jobrunner-otel-collector_1      |      -> host.arch: Str(amd64)
jobrunner-otel-collector_1      |      -> host.name: Str(3973966180a2)
jobrunner-otel-collector_1      |      -> os.description: Str(Linux 5.15.123.1-microsoft-standard-WSL2)
jobrunner-otel-collector_1      |      -> os.type: Str(linux)
jobrunner-otel-collector_1      |      -> process.command_line: Str(/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -javaagent:/opentelemetry-javaagent.jar app.jar --spring.profiles.active=localdev java -javaagent:/opentelemetry-javaagent.jar -jar /app.jar --spring.profiles.active=localdev)
jobrunner-otel-collector_1      |      -> process.executable.path: Str(/usr/lib/jvm/java-1.8-openjdk/jre/bin/java)
jobrunner-otel-collector_1      |      -> process.pid: Int(1)
jobrunner-otel-collector_1      |      -> process.runtime.description: Str(IcedTea OpenJDK 64-Bit Server VM 25.212-b04)
jobrunner-otel-collector_1      |      -> process.runtime.name: Str(OpenJDK Runtime Environment)
jobrunner-otel-collector_1      |      -> process.runtime.version: Str(1.8.0_212-b04)
jobrunner-otel-collector_1      |      -> service.name: Str(jobrunner)
jobrunner-otel-collector_1      |      -> telemetry.auto.version: Str(1.30.0)
jobrunner-otel-collector_1      |      -> telemetry.sdk.language: Str(java)
jobrunner-otel-collector_1      |      -> telemetry.sdk.name: Str(opentelemetry)
jobrunner-otel-collector_1      |      -> telemetry.sdk.version: Str(1.30.1)
jobrunner-otel-collector_1      | ScopeMetrics #0
jobrunner-otel-collector_1      | ScopeMetrics SchemaURL:
jobrunner-otel-collector_1      | InstrumentationScope spanmetricsconnector
jobrunner-otel-collector_1      | Metric #0
jobrunner-otel-collector_1      | Descriptor:
jobrunner-otel-collector_1      |      -> Name: calls
jobrunner-otel-collector_1      |      -> Description:
jobrunner-otel-collector_1      |      -> Unit:
jobrunner-otel-collector_1      |      -> DataType: Sum
jobrunner-otel-collector_1      |      -> IsMonotonic: true
jobrunner-otel-collector_1      |      -> AggregationTemporality: Cumulative
jobrunner-otel-collector_1      | NumberDataPoints #0
jobrunner-otel-collector_1      | Data point attributes:
jobrunner-otel-collector_1      |      -> service.name: Str(jobrunner)
jobrunner-otel-collector_1      |      -> span.name: Str(GET /monet/influx/version)
jobrunner-otel-collector_1      |      -> span.kind: Str(SPAN_KIND_SERVER)
jobrunner-otel-collector_1      |      -> status.code: Str(STATUS_CODE_UNSET)
jobrunner-otel-collector_1      |      -> http.method: Str(GET)
jobrunner-otel-collector_1      |      -> http.status_code: Int(200)
jobrunner-otel-collector_1      |      -> http.route: Str(/monet/influx/version)
jobrunner-otel-collector_1      | StartTimestamp: 2023-11-01 19:13:29.039592939 +0000 UTC
jobrunner-otel-collector_1      | Timestamp: 2023-11-01 19:16:29.040404546 +0000 UTC
jobrunner-otel-collector_1      | Value: 2
jobrunner-otel-collector_1      | Metric #1
jobrunner-otel-collector_1      | Descriptor:
jobrunner-otel-collector_1      |      -> Name: duration
jobrunner-otel-collector_1      |      -> Description:
jobrunner-otel-collector_1      |      -> Unit: ms
jobrunner-otel-collector_1      |      -> DataType: Histogram
jobrunner-otel-collector_1      |      -> AggregationTemporality: Cumulative
jobrunner-otel-collector_1      | HistogramDataPoints #0
jobrunner-otel-collector_1      | Data point attributes:
jobrunner-otel-collector_1      |      -> service.name: Str(jobrunner)
jobrunner-otel-collector_1      |      -> span.name: Str(GET /monet/influx/version)
jobrunner-otel-collector_1      |      -> span.kind: Str(SPAN_KIND_SERVER)
jobrunner-otel-collector_1      |      -> status.code: Str(STATUS_CODE_UNSET)
jobrunner-otel-collector_1      |      -> http.method: Str(GET)
jobrunner-otel-collector_1      |      -> http.status_code: Int(200)
jobrunner-otel-collector_1      |      -> http.route: Str(/monet/influx/version)
jobrunner-otel-collector_1      | StartTimestamp: 2023-11-01 19:13:29.039592939 +0000 UTC
jobrunner-otel-collector_1      | Timestamp: 2023-11-01 19:16:29.040409646 +0000 UTC
jobrunner-otel-collector_1      | Count: 2
jobrunner-otel-collector_1      | Sum: 1817.928041
jobrunner-otel-collector_1      | ExplicitBounds #0: 2.000000
jobrunner-otel-collector_1      | ExplicitBounds #1: 4.000000
jobrunner-otel-collector_1      | ExplicitBounds #2: 6.000000
jobrunner-otel-collector_1      | ExplicitBounds #3: 8.000000
jobrunner-otel-collector_1      | ExplicitBounds #4: 10.000000
jobrunner-otel-collector_1      | ExplicitBounds #5: 50.000000
jobrunner-otel-collector_1      | ExplicitBounds #6: 100.000000
jobrunner-otel-collector_1      | ExplicitBounds #7: 200.000000
jobrunner-otel-collector_1      | ExplicitBounds #8: 400.000000
jobrunner-otel-collector_1      | ExplicitBounds #9: 800.000000
jobrunner-otel-collector_1      | ExplicitBounds #10: 1000.000000
jobrunner-otel-collector_1      | ExplicitBounds #11: 1400.000000
jobrunner-otel-collector_1      | ExplicitBounds #12: 2000.000000
jobrunner-otel-collector_1      | ExplicitBounds #13: 5000.000000
jobrunner-otel-collector_1      | ExplicitBounds #14: 10000.000000
jobrunner-otel-collector_1      | ExplicitBounds #15: 15000.000000
jobrunner-otel-collector_1      | Buckets #0, Count: 0
jobrunner-otel-collector_1      | Buckets #1, Count: 0
jobrunner-otel-collector_1      | Buckets #2, Count: 0
jobrunner-otel-collector_1      | Buckets #3, Count: 0
jobrunner-otel-collector_1      | Buckets #4, Count: 0
jobrunner-otel-collector_1      | Buckets #5, Count: 0
jobrunner-otel-collector_1      | Buckets #6, Count: 0
jobrunner-otel-collector_1      | Buckets #7, Count: 0
jobrunner-otel-collector_1      | Buckets #8, Count: 1
jobrunner-otel-collector_1      | Buckets #9, Count: 0
jobrunner-otel-collector_1      | Buckets #10, Count: 0
jobrunner-otel-collector_1      | Buckets #11, Count: 0
jobrunner-otel-collector_1      | Buckets #12, Count: 1
jobrunner-otel-collector_1      | Buckets #13, Count: 0
jobrunner-otel-collector_1      | Buckets #14, Count: 0
jobrunner-otel-collector_1      | Buckets #15, Count: 0
jobrunner-otel-collector_1      | Buckets #16, Count: 0
jobrunner-otel-collector_1      | Exemplars:
jobrunner-otel-collector_1      | Exemplar #0
jobrunner-otel-collector_1      |      -> Trace ID: 249460d4c98040997d836f00a190c7da
jobrunner-otel-collector_1      |      -> Span ID: 20f8de9b97fbe1d8
jobrunner-otel-collector_1      |      -> Timestamp: 2023-11-01 19:16:29.040409646 +0000 UTC
jobrunner-otel-collector_1      |      -> Value: 311.627870
jobrunner-otel-collector_1      |       {"kind": "exporter", "data_type": "metrics", "name": "logging"}


Response from scrape endpoint:

# HELP calls 
# TYPE calls counter
calls{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET"} 6
# HELP duration 
# TYPE duration histogram
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="2"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="4"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="6"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="8"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="10"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="50"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="100"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="200"} 0
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="400"} 5
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="800"} 5
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="1000"} 5
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="1400"} 5
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="2000"} 6
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="5000"} 6
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="10000"} 6
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="15000"} 6
duration_bucket{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET",le="+Inf"} 6
duration_sum{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET"} 3022.686396
duration_count{http_method="GET",http_route="/monet/influx/version",http_status_code="200",job="jobrunner",service_name="jobrunner",span_kind="SPAN_KIND_SERVER",span_name="GET /monet/influx/version",status_code="STATUS_CODE_UNSET"} 6

Additional context

No response
@ctrader2 ctrader2 added bug Something isn't working needs triage New item requiring triage labels Nov 1, 2023
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 1, 2023

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@ctrader2 ctrader2 mentioned this issue Nov 1, 2023
Closed
@yantingqiu
Copy link

I would suggest giving the following configuration a try.

exporters:
  otlp:
    endpoint: grafana-agent:4317
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
    enable_open_metrics: true
    add_metric_suffixes: false
    resource_to_telemetry_conversion: true   //  If `enabled` is `true`, all the resource attributes will be converted to metric labels by default.

@github-actions github-actions bot mentioned this issue Nov 7, 2023
Closed
@crobert-1
Copy link
Member

@ctrader2 have you been able to test @yantingqiu's shared configuration?

Note: I believe the correct configuration for this option would be:

resource_to_telemetry_conversion:
  enabled: true

@github-actions github-actions bot mentioned this issue Nov 14, 2023
Closed
This was referenced Nov 21, 2023
Closed
Closed
@github-actions github-actions bot mentioned this issue Dec 5, 2023
Closed
@crobert-1
Copy link
Member

I'm going to close for now as I believe the suggested configuration update will resolve the issue. Please let us know though if you get a chance to test and that's not the case!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working connector/spanmetrics exporter/prometheus needs triage New item requiring triage waiting for author
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ctrader2 @yantingqiu @crobert-1