Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve OTel error logging #2572

Merged
merged 1 commit into from
Jun 20, 2024

Conversation

calebschoepp
Copy link
Collaborator

Closes #2565

Opted to rate limit all errors and not just channel full errors.

@calebschoepp calebschoepp requested a review from lann June 17, 2024 21:27
crates/telemetry/src/lib.rs Outdated Show resolved Hide resolved
crates/telemetry/src/lib.rs Outdated Show resolved Hide resolved
crates/telemetry/src/lib.rs Outdated Show resolved Hide resolved
Copy link
Collaborator

@lann lann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opted to rate limit all errors and not just channel full errors.

It occurs to me that this could hide important errors among noise... maybe that's OK? 🤔

crates/telemetry/src/lib.rs Outdated Show resolved Hide resolved
@calebschoepp
Copy link
Collaborator Author

It occurs to me that this could hide important errors among noise... maybe that's OK? 🤔

Probably just okay. If it is something other than a channel full error it will likely show up as the first error. And at the very least we'll always be tracking it at trace.

Copy link
Collaborator

@lann lann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh everything currently unresolved is non-blocking

Signed-off-by: Caleb Schoepp <[email protected]>
@calebschoepp calebschoepp merged commit 7ef8d8f into fermyon:main Jun 20, 2024
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OTel features overwhelmed during high load
2 participants