Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OOM error caused by AzureMonitorLogRecordExporter #40546

Closed
rafalp88 opened this issue Jun 7, 2024 · 2 comments · Fixed by #40615
Closed

OOM error caused by AzureMonitorLogRecordExporter #40546

rafalp88 opened this issue Jun 7, 2024 · 2 comments · Fixed by #40615
Assignees
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Monitor - Exporter Monitor OpenTelemetry Exporter needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@rafalp88
Copy link

rafalp88 commented Jun 7, 2024

Below line cause OOM error. When DEBUG level is on, there will be some kind of recursive invocation.
LogRecords logging causes more log entries which causes sending more and more LogRecords.

@github-actions github-actions bot added Azure.Core azure-core Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jun 7, 2024
Copy link

github-actions bot commented Jun 7, 2024

Thank you for your feedback. Tagging and routing to the team member best able to assist.

@lmolkova lmolkova assigned lmolkova and unassigned alzimmermsft Jun 7, 2024
@lmolkova
Copy link
Member

lmolkova commented Jun 11, 2024

@rafalp88

Thank you for reporting this!
As a mitigation step, could you please disable verbose logs from com.azure.monitor.opentelemetry.exporter? If you haven't done it already.

@heyams @trask could you please take a look?

it seems exporter logs with ClientLogger from core which, when such debug logs are enabled, produces enormous amount of logs.

We might need to figure out how to disable all logs from the exporter http pipeline completely or prevent the exporter from consuming its own logs in some other way. LMK if you need any help with ClientLogger.

@lmolkova lmolkova added Monitor - Exporter Monitor OpenTelemetry Exporter and removed Azure.Core azure-core labels Jul 15, 2024
@lmolkova lmolkova removed their assignment Jul 15, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Oct 22, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. Monitor - Exporter Monitor OpenTelemetry Exporter needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

5 participants