[BUG] Azure.Core OpenTelemetry breaks packages

[martijnmelchers]

Library name and version

Azure.Core 1.36.0

Describe the bug

We use multiple packages using Azure.Core after the latest update to 1.36.0 the CertificateClient broke

It seems to be related to #39617

More specifically this?
https://github.com/Azure/azure-sdk-for-net/pull/39617/files#diff-8b0ecb439ab5c789e43df6453a01bd0fc375f00c8844b266491c53062620de33R130

Expected behavior

Start without issues

Actual behavior

It doesn't start and throws the following error:

Exception Info: System.ArgumentException: An item with the same key has already been added. Key: error.type
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.AddRange(IEnumerable`1 collection)
   at Elastic.Apm.OpenTelemetry.ElasticActivityListener.<get_ActivityStopped>b__21_0(Activity activity)
   at System.Diagnostics.SynchronizedList`1.EnumWithAction(Action`2 action, Object arg)
   at System.Diagnostics.Activity.Stop()
   at System.Diagnostics.Activity.Dispose()
   at Azure.Core.Pipeline.DiagnosticScope.ActivityAdapter.Dispose()
   at Azure.Core.Pipeline.DiagnosticScope.Dispose()
   at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
   at Azure.Core.Pipeline.RequestActivityPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
   at Azure.Core.Pipeline.ResponseBodyPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.LoggingPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
   at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
   at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.HttpPipeline.SendRequest(Request request, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.KeyVaultPipeline.SendRequest(Request request, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.KeyVaultPipeline.SendRequest[TResult](RequestMethod method, Func`1 resultFactory, CancellationToken cancellationToken, String[] path)
   at Azure.Security.KeyVault.Certificates.CertificateClient.DownloadCertificate(DownloadCertificateOptions options, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.Certificates.CertificateClient.DownloadCertificate(String certificateName, String version, CancellationToken cancellationToken)
   at Api.Extensions.StartupExtensions.AddRavenStore(IServiceCollection services, IConfiguration configuration, IHostEnvironment environment) in /home/runner/work/ngSneleentaxi/ngSneleentaxi/src/Api/Extensions/StartupExtensions.cs:line 311
   at Microsoft.Extensions.DependencyInjection.ConfigureServices.AddApiServices(IServiceCollection services, IConfiguration config, IHostEnvironment environment) in /home/runner/work/<>/<>/src/Api/ConfigureServices.cs:line 45
   at Api.Program.Main(String[] args) in /home/runner/work/<>/<>/src/Api/Program.cs:line 49
   at Api.Program.&lt;Main&gt;(String[] args)

Reproduction Steps

Use Azure.Security.KeyVault.Certificates 4.5.1 with Azure.Core 1.36.0 and do something like this:

var client = new CertificateClient(new Uri(<uri>), new DefaultAzureCredential());
cert = client.DownloadCertificate(<cert>);

Environment

.NET SDK:
Version: 7.0.403
Commit: 142776d834

Runtime Environment:
OS Name: Windows
OS Version: 10.0.22621
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\7.0.403\

[jsquire]

@lmolkova: Your thoughts would be appreciated.

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[lmolkova]

@martijnmelchers thank you for reporting this and sorry it breaks!

It's been fixed in #40027, but the fix is not released yet and likely to be shipped in January.

Could you please create a bug for Elastic APM on this? Activity.TagObjects are not guaranteed to be unique and this code will keep breaking in other cases:

   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.AddRange(IEnumerable`1 collection)
   at Elastic.Apm.OpenTelemetry.ElasticActivityListener.<get_ActivityStopped>b__21_0(Activity activity)

[martijnmelchers]

I'll see if I can report this, however this now means we can't bump packages with fixes and such, could a patch be created so we don't have to wait two months for this fix?

[lmolkova]

We're considering shipping a hotfix, but for now let's see if we can unblock you right away.

One option would be to change how you enable Azure SDK tracing - the problem only affects Azure.Core.Http ActivitySource and you don't really have to enable it.

You can get almost the same level of details if you use generic HTTP client instrumentation. Please check out our docs on how to enable activity sources for individual libraries

Let me know if it helps.

[martijnmelchers]

We've currently downgraded back a patch. Its unfortunate because this change came shipped as a patch. I'll try this later if we really need it, thanks!

[JoshLove-msft]

The fix for this issue has shipped as part of Azure.Core 1.37.0.