Backport #61798 to .NET 6

[madelson]

#61798 is breaking concurrent use of the .NET Azure SDKs for us starting in .NET 6 (see Azure/azure-sdk-for-net#27018, for example). Can this fix be backported to .NET 6?

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

#61798 is breaking concurrent use of the .NET Azure SDKs for us starting in .NET 6 (see Azure/azure-sdk-for-net#27018, for example). Can this fix be backported to .NET 6?

Author:	madelson
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[MihaZupan]

Judging by the stack trace you posted in Azure/azure-sdk-for-net#27018 (comment), you appear to be hitting #65379.

Can you please try the workaround mentioned in the issue?

[MihaZupan]

Duplicate of #65379.

(Of course, feel free to continue discussion around this issue there)

[madelson]

@MihaZupan thanks for the info on the other error.

However, I'm also seeing the index-out-of-bounds error (see Azure/azure-sdk-for-net#27018 (comment)) so I'd still appreciate that fix being backported.

[MihaZupan]

#65379 is the cause for that IndexOutOfRangeException (see EstablishProxyTunnelAsync in the stack trace).

[madelson]

@MihaZupan ah ok I misunderstood. So #61799 covers both stacktraces and #68115 is completely unrelated?:

System.IndexOutOfRangeException: Index was outside the bounds of the array. 
at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex) 
at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue) 
at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringArray(HeaderDescriptor descriptor, HeaderStoreItemInfo info) 
at System.Net.Http.Headers.HttpHeaders.TryGetValues(HeaderDescriptor descriptor, IEnumerable`1& values) 
at System.Net.Http.HttpConnectionPool.EstablishProxyTunnelAsync(Boolean async, HttpRequestHeaders headers, CancellationToken cancellationToken) 
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) 
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) 
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) 
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) 
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) 
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) 
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) 
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) 
at System.Net.Http.HttpClient.<SendAsync>g_Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) 

AND

System.InvalidOperationException: Collection was modified; enumeration operation may not execute. 
at System.Collections.Generic.List`1.Enumerator.MoveNextRare() 
at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex)
at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue) 
at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringArray(HeaderDescriptor descriptor, HeaderStoreItemInfo info) 
at System.Net.Http.HttpConnectionPool.EstablishProxyTunnelAsync(Boolean async, HttpRequestHeaders headers, CancellationToken cancellationToken) 
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) 
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) 
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) 
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) 
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) 
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) 
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) 
at System.Net.Http.HttpClient.<SendAsync>g_Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) 

Is that correct?

[MihaZupan]

#61798 tracks the issue that before 7.0, reading from a header collection concurrently was not thread-safe, blowing up in many different ways (like the two exceptions you observed).

We treat concurrent use like that as improper usage. But there are cases where HttpClient itself accesses the collection when it shouldn't, leading to errors in otherwise well-behaved client code.
#65379 tracks one such bug that occurs if the user uses a proxy. In your case, both exceptions are just different manifestations of the same race condition.

TL;DR; #65379 covers both of the exceptions you observed.
#68115 is not completely unrelated - it mitigates the issue, but doesn't eliminate it completely.

[madelson]

@MihaZupan thanks. It is still my understanding that these problems are unique to (or at least much more prevalent in) .NET 6; we're hitting them through the Azure SDK and we did not see them prior to the .NET 6 upgrade. Do you think any of the relevant changes will be backported to 6?