[FEATURE REQ] Very difficult to know what went wrong when posts to Azure Monitor fail

[samsp-msft]

Library name

Azure.Monitor.OpenTelemetry.Exporter

Please describe the feature.

Diagnosing why the OpenTelemetry SDK is not working is too hard and un-obvious to non-team members.

Currently errors in reporting are logged as EventSource events, and unbeknownst to me, in the Visual Studio output window. This needs to be made much more obvious to developers. I was running dotnet-monitor and having to do a post to configure the event collection as it's not a standard source, and having to break in the debugger to enable the collection before some of the events are fired.

curl -X 'POST' \
  'https://localhost:52323/trace?name=loadtest&durationSeconds=30' \
  -H 'accept: application/octet-stream' \
  -H 'Content-Type: application/json' \
  -d '{
  "providers": [
    {"name": "OpenTelemetry-Sdk","eventLevel": "5"},
    {"name": "OpenTelemetry-AzureMonitor-Exporter","eventLevel": "5"},
    {"name": "Azure-Core","5": "Debug"}
  ],
  "requestRundown": true,
  "bufferSizeInMB": 1024
}'

When running ASP.NET Core Apps, typically I look to the console output to tell me what is going wrong. If Open Telemetry is failing, it should probably be using ILogger to write output to the console, and use appropriate levels so only error information is included by default (that can be changed through config).

Similarly, when I have problems with the deployed app, the first place to look is the console. Azure App-Service, Functions and Container Apps all have support for streaming the console output. So reporting to the console is the first place to look for explanations of issues.

I tried using OTEL_DIAGNOSTICS.json - and the file just contained Successfully opened file.

[samsp-msft]

@reyang @bradygaster

[samsp-msft]

@cijothomas

[cijothomas]

https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/README.md#self-diagnostics

If you enabled "Error" level, and nothing of Error severity has occurred then that file will be basically empty (apart from the opened file message..TODO: I think this message should be improved to call out that...)

Could you make it Verbose?

Anyway, if nothing of Error severity has occurred, then its a good indication everything is working well. What was the issue you were having that you attempted to troubleshoot?

[samsp-msft]

I put a bad connection string, so I am getting a 400 when logs are posted.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas, @reyang, @rajkumar-rangaraj, @TimothyMothra, @vishweshbankwar.

Issue Details

---

Library name

Azure.Monitor.OpenTelemetry.Exporter

Please describe the feature.

Diagnosing why the OpenTelemetry SDK is not working is too hard and un-obvious to non-team members.

Currently errors in reporting are logged as EventSource events, and unbeknownst to me, in the Visual Studio output window. This needs to be made much more obvious to developers. I was running dotnet-monitor and having to do a post to configure the event collection as it's not a standard source, and having to break in the debugger to enable the collection before some of the events are fired.

curl -X 'POST' \
  'https://localhost:52323/trace?name=loadtest&durationSeconds=30' \
  -H 'accept: application/octet-stream' \
  -H 'Content-Type: application/json' \
  -d '{
  "providers": [
    {"name": "OpenTelemetry-Sdk","eventLevel": "5"},
    {"name": "OpenTelemetry-AzureMonitor-Exporter","eventLevel": "5"},
    {"name": "Azure-Core","5": "Debug"}
  ],
  "requestRundown": true,
  "bufferSizeInMB": 1024
}'

When running ASP.NET Core Apps, typically I look to the console output to tell me what is going wrong. If Open Telemetry is failing, it should probably be using ILogger to write output to the console, and use appropriate levels so only error information is included by default (that can be changed through config).

Similarly, when I have problems with the deployed app, the first place to look is the console. Azure App-Service, Functions and Container Apps all have support for streaming the console output. So reporting to the console is the first place to look for explanations of issues.

I tried using OTEL_DIAGNOSTICS.json - and the file just contained Successfully opened file.

Author:	samsp-msft
Assignees:	-
Labels:	Service Attention, Client, customer-reported, question, needs-team-attention, Monitor - Exporter
Milestone:	-

[cijothomas]

I put a bad connection string, so I am getting a 400 when logs are posted.

If you mean the connectionstring was malformatted/unparseable, I'd expect AzMon exporter to throw and let app crash on startup.
@vishweshbankwar - Please confirm if the above is correct.

If you mean your connectionstring was correct, but it had wrong ikey or wrong endpoint, then I'd expect AzMon Exporter to log the fact that it received failure (400) and optionally return "ExportFailed" back to the OTel SDK (which would log it).

@vishweshbankwar - Please confirm if Exporter is doing this logging or not.

[TimothyMothra]

Multiple points to respond to :)

1. If you mean the connectionstring was malformatted/unparseable, I'd expect AzMon exporter to throw and let app crash on startup.

   This is correct. If the connectionstring is malformed or if the InstrumentationKey is not present, the exporter will throw/crash with a detailed message.

2. If you mean your connectionstring was correct, but it had wrong ikey or wrong endpoint, then I'd expect AzMon Exporter to log the fact that it received failure (400)

   This is correct, we log these to Event Source which is also visible in Visual Studio's Debug Output:
   

   The ingestion sometimes responds with a detailed error message. We should double check that we're logging those messages!

3. and optionally return "ExportFailed" back to the OTel SDK (which would log it).

   This is correct. Our Transmitter will return ExportResult.Failed.
   Regarding OTel's Self Diagnostic behavior, I confirmed this is capturing logs from our EventSource.
   
   

4. When running ASP.NET Core Apps, typically I look to the console output to tell me what is going wrong. If Open Telemetry is failing, it should probably be using ILogger to write output to the console, and use appropriate levels so only error information is included by default (that can be changed through config).

   This is a good idea but is a major change from what we and the community are doing today.
   I've seen similar asks in the https://github.com/open-telemetry/opentelemetry-dotnet repo but I haven't seen any consensus to move away from EventSource.
   Perhaps we can pilot this as a feature in the AzureMonitorExporter and share learnings with the community.

[reyang]

When running ASP.NET Core Apps, typically I look to the console output to tell me what is going wrong. If Open Telemetry is failing, it should probably be using ILogger to write output to the console, and use appropriate levels so only error information is included by default (that can be changed through config).

OpenTelemetry .NET as a general purpose library, it cannot write to console by default since this would break the user's expectation (especially if the user is expecting certain console output format). It is possible to allow the internal error logs to be configured to output to the console (e.g. it'll be even better if the internal logs are exposed via ILogger so we have the flexibility to write to file/console/etc.).

@samsp-msft Is there a contract between hosting environments (e.g. Functions) and libraries (e.g. OTel.NET) so libraries can expose the logs that hosting environments could subscribe to?

[samsp-msft]

AFAIK the only service that has a specific contract around logging is Functions as they pass you an ILogger instance as part of the entrypoint into your function.

The various flavors of app-service don't have a specific contract with .NET around logging. They primarily rely on providing environment variables for app-insights. They have the ability to view the console output from the process that they are hosting via a Log Stream blade.

Container Apps relies on console output as its built-in logging mechanism, and provides a log stream, and collection of those entries into an azure monitor table configured as part of creation of the app.

There seems to be some precedent for directly outputting to the console from OTEL - that seems to be what using .AddConsoleExporter() is doing.

[cijothomas]

There seems to be some precedent for directly outputting to the console from OTEL - that seems to be what using .AddConsoleExporter() is doing.

^ Is only for learning purposes. It emits the actual telemetry itself, not the SDK internal logs. Not for production usage.

[cijothomas]

e.g. it'll be even better if the internal logs are exposed via ILogger so we have the flexibility to write to file/console/etc.

Yes, this is the tracking issue for that : open-telemetry/opentelemetry-dotnet#3881
(surprisingly not many upvotes/comments there!)

[TimothyMothra]

I'm going to close this issue.
We rewrote all diagnostic logging messages to make them more informative or actionable.