Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Activity.RecordException now correctly updates the error status of OpenTelemetry Spans #2515

Merged
merged 6 commits into from
Jul 27, 2023

Conversation

jamescrosswell
Copy link
Collaborator

@jamescrosswell jamescrosswell commented Jul 27, 2023

Fixes #2514

Problem

When users of our OpenTelemetry integration catch errors and record these using Activity.RecordException (vs using SentrySdk.CatpureException) the span that gets created in Sentry is not correctly categorised as an error span and doesn't have stack trace information etc.

Solution

When exceptions capturing is instrumented this way, the OpenTelemetry .NET SDK currently doesn't provide access to the original exception, so we can't get as much information as we could via SentrySdk.CatpureException but we can recreate a synthetic exception that includes some of the original exception details and ensures the Span that we record in Sentry shows correctly as errored.

The OpenTelemetry .NET SDK provides us with the exception type, message and a "stack trace" (which is Exception.ToString()) that might look something like this:

Program+MyException: Attempted to divide by zero.
    ---> System.DivideByZeroException: Attempted to divide by zero.
    at Program.Main()
    --- End of inner exception stack trace ---
    at Program.Main()   

For something like an AggregateException that stack trace would look more complicated.

It might be possible to parse this into an exception graph that could be then be used in our MainExceptionProcessor to create one or more SentryExceptions. However it probably makes more sense to push to get the underlying issue in the Otel codebase resolved instead.

For the time being, we're just putting the stack trace provided by OpenTelemetry in an otel.stack_trace attribute that gets sent with the event for the exception. So people can see the stack trace provided by OpenTelemetry if they want.

@jamescrosswell jamescrosswell marked this pull request as ready for review July 27, 2023 08:54
@jamescrosswell jamescrosswell changed the title Fixed Span Status when using OTEL Activity.SetStatus Fixed Error status of OpenTelemetry Spans when using Activity.RecordException Jul 27, 2023
Comment on lines +333 to +336
// TODO: Validate that our `DuplicateEventDetectionEventProcessor` prevents this from doubling exceptions
// that are also caught by other means, such as our AspNetCore middleware, etc.
// (When options.RecordException = true is set on AddAspNetCoreInstrumentation...)
// Also, in such cases - how will we get the otel scope and trace context on the other one?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR looks good to me. The TODO here sounds like quite the gotcha.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does. It occurs if you do something like this:

        using var activity = Telemetry.ActivitySource.StartActivity("Funny business");
        try
        {
            throw new Exception("test activity.RecordException");
        }
        catch (Exception e)
        {
            SentrySdk.CaptureException(e);
            activity?.RecordException(e);
            activity?.SetStatus(Status.Error.WithDescription(e.Message));
        }
        return "Catch";

The deduplication doesn't work at all (since it matches by reference). Again, we could come up with convoluted ways of working around this but I think the ultimate solution is to address the issue in the OpenTelemetry codebase rather than try to work around it in ours.

I'll create an issue for the above in our repository saying as much though.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHANGELOG.md Outdated Show resolved Hide resolved
Co-authored-by: Stefan Jandl <[email protected]>
@jamescrosswell jamescrosswell merged commit d4392e2 into main Jul 27, 2023
14 checks passed
@jamescrosswell jamescrosswell deleted the fix/2514-otel-error-status branch July 27, 2023 21:47
@jamescrosswell jamescrosswell changed the title Fixed Error status of OpenTelemetry Spans when using Activity.RecordException Using Activity.RecordException now correctly updates the error status of OpenTelemetry Spans Jul 27, 2023
@github-actions
Copy link
Contributor

Fails
🚫 Please consider adding a changelog entry for the next release.

Instructions and example for changelog

Please add an entry to CHANGELOG.md to the "Unreleased" section. Make sure the entry includes this PR's number.

Example:

## Unreleased

- Using `Activity.RecordException` now correctly updates the error status of OpenTelemetry Spans ([#2515](https://github.com/getsentry/sentry-dotnet/pull/2515))

If none of the above apply, you can opt out of this check by adding #skip-changelog to the PR description.

Generated by 🚫 dangerJS against 38ae2bf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error status of OpenTelemetry Spans is not being correctly reflected in Sentry spans.
2 participants