Spans misclassified as child span when app is deployed to an Azure app service

[bradleyberwick]

I have instrumented OTEL tracing in my .Net app with with the below OTEL tracing instrumentations. When I run it locally and hit an endpoint, a span is generated and is correctly classified as a parent span. However, when I deploy the app to an Azure app service, a span is generated with no parent span ID i.e. this span is misclassified as a parent span.

Does anyone know what might be causing this issue in Azure?

• .AddAspNetCoreInstrumentation()
• .AddHttpClientInstrumentation()

[vishweshbankwar]

@bradleyberwick - Could you share your SDK configuration?

[bradleyberwick]

@vishweshbankwar here's a snippet of our .Net startup file:

services.AddOpenTelemetry() .WithTracing(tracerBuilder => tracerBuilder .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(_config["Otlp:ServiceName"])) .AddAspNetCoreInstrumentation() .AddGrpcClientInstrumentation() .AddHttpClientInstrumentation() .AddSqlClientInstrumentation() .SetSampler(new AlwaysOnSampler()) //.AddConsoleExporter() // for debugging purpose .AddOtlpExporter(opt => { opt.Endpoint = new Uri(_config["Otlp:Endpoint"]); })) .WithMetrics(meterBuilder => meterBuilder .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(_config["Otlp:ServiceName"])) .AddProcessInstrumentation() .AddRuntimeInstrumentation() .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() //.AddConsoleExporter() // for debugging purpose .AddOtlpExporter(opt => { opt.Endpoint = new Uri(_config["Otlp:Endpoint"]); }));

[bradleyberwick]

@vishweshbankwar we see the same amount of spans locally vs app service. The span for the app service for some reason thinks there should be another span (i.e. the parent span) which as far as I know doesn't exist. Can you give me an example of what you mean by output snippet please?

[vishweshbankwar]

@bradleyberwick

we see the same amount of spans locally vs app service. The span for the app service for some reason thinks there should be another span (i.e. the parent span) which as far as I know doesn't exist

This part is not clear to me. An output from console exporter locally would work and you can point what changes when you deploy to app service.

[bradleyberwick]

@vishweshbankwar Below is an example from the console exporter running the app locally. Like I said before, when running the app locally, OTEL correctly doesn't provide a ParentSpanId value. However, when the app is deployed in Azure, OTEL incorrectly adds a ParentSpanId, however I can't see said span ID or where it is being generated from.

Activity.TraceId: d3f4ee049250069024477bf314dbca0b Activity.SpanId: 4cca7bfcd6f5cdc9 Activity.TraceFlags: Recorded Activity.ActivitySourceName: OpenTelemetry.Instrumentation.AspNetCore Activity.ActivitySourceVersion: 1.0.0.0 Activity.DisplayName: GET /media/y3uhjsoy/sc-3_pots_of_gold_land_and_win_thumb_non-gotw.jpg Activity.Kind: Server Activity.StartTime: 2024-08-06T08:37:46.9663959Z Activity.Duration: 00:00:00.9431233 Activity.Tags: server.address: localhost server.port: 65510 http.request.method: GET url.scheme: https url.path: /media/y3uhjsoy/sc-3_pots_of_gold_land_and_win_thumb_non-gotw.jpg network.protocol.version: 2 user_agent.original: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 http.response.status_code: 200 http.route: GET /media/y3uhjsoy/sc-3_pots_of_gold_land_and_win_thumb_non-gotw.jpg Resource associated with Activity: service.name: h20-cms-local service.instance.id: c09dfa15-4bdf-4260-8485-c139d7c3196b telemetry.sdk.name: opentelemetry telemetry.sdk.language: dotnet telemetry.sdk.version: 1.8.1

[vishweshbankwar]

@bradleyberwick - That means the component that's calling your API on Azure is passing in the traceparent header. Do you know which component sends the request?

[bradleyberwick]

@vishweshbankwar The problem is though when you call the API directly (e.g. hitting the ...azurewebsites.net url in browser/postman), where no traceheader is explicitly added, we still get the same issue. I don't understand where the traceheader is getting added from.