System.Net test suites asserts in CI on ArrayBuffer.Discard

[ViktorHofer]

Failures 7/30-8/19 (incl. PRs)

Day	Run	OS	Notes
8/2	PR #56723	OSX.1100.ARM64.Open	System.Net.Http.Functional.Tests
8/11	PR #57217	OSX.1014.Amd64.Open	System.Net.Security.Tests

Configuration: net6.0-OSX-Debug-x64-CoreCLR_release-OSX.1015.Amd64.Open

Build:
https://dev.azure.com/dnceng/public/_build/results?buildId=861070&view=ms.vss-test-web.build-test-results-tab&runId=27582874&paneView=debug

console.df11c71b.log:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-43583-merge-76feeba0b9b24d0cac/System.Net.Security.Tests/console.df11c71b.log?sv=2019-07-07&se=2020-11-10T12%3A04%3A15Z&sr=c&sp=rl&sig=ollbotIg4oFm3EfLfaPrGBm4k92qWZ%2BrhXS7ydWvQ0I%3D

Process terminated. Assertion failed.
Expected 206 <= 0
   at System.Net.ArrayBuffer.Discard(Int32 byteCount) in /_/src/libraries/Common/src/System/Net/ArrayBuffer.cs:line 71
   at System.Net.SafeDeleteSslContext.ReadPendingWrites() in /_/src/libraries/System.Net.Security/src/System/Net/Security/Pal.OSX/SafeDeleteSslContext.cs:line 219
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.OSX.cs:line 258
   at System.Net.Security.SslStreamPal.InitializeSecurityContext(SafeFreeCredentials& credential, SafeDeleteSslContext& context, String targetName, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.OSX.cs:line 52
   at System.Net.Security.SecureChannel.GenerateToken(ReadOnlySpan`1 inputBuffer, Byte[]& output) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 803
   at System.Net.Security.SecureChannel.NextMessage(ReadOnlySpan`1 incomingBuffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 731
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 327
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 42
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 215
   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 297
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 283
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 264
   at System.Net.Security.Tests.SslStreamSniTest.<>c__DisplayClass1_1.<SslStream_ServerCallbackAndLocalCertificateSelectionSet_Throws>b__2() in /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSniTest.cs:line 78
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 274
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2321
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 641

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[ViktorHofer]

cc @dotnet/ncl

[ViktorHofer]

@danmosemsft @maryamariyan see the bot's message above. The @dotnet/ncl comment is not highlighted as a mention.

[stephentoub]

cc: @wfurt

[wfurt]

Did we get core file @ViktorHofer ?

[ViktorHofer]

Unfortunately no.

[wfurt]

how so? OSX core files are pain to deal with, but I thought we had that in place. I have some suspicions but that would be hard to verify without core or repro. I can do loop test run to see if I can make it happen locally.

From the trace, it looks like the offending test was SslStream_ServerCallbackAndLocalCertificateSelectionSet_Throws.
The client side starts as

                Task clientJob = Task.Run(() => {
                    client.AuthenticateAsClient(hostName);
                    Assert.True(false, "RemoteCertificateValidationCallback called when AuthenticateAsServerAsync was expected to fail.");
                });

inside of using statement. So I assume the client can be disposed while running as soon as the server fails, right @stephentoub?
I'm also wondering if it would be useful to add tests to dispose in the middle of various SslStream operations. It seems like our current SslStreamDisposeTest is pretty trivial.

[stephentoub]

So I assume the client can be disposed while running as soon as the server fails, right @stephentoub?

This test is poorly written. I'm not sure what that's trying to validate, but the Task returned from the Task.Run is never joined with, so any exceptions thrown from AuthenticateAsClient or from a failing assert are just going to be eaten. And as you say, assuming the server call can fail before or part-way through the client authenticating, the client can be disposed of during this synchronous AuthenticateAsClient call.

[ViktorHofer]

how so? OSX core files are pain to deal with, but I thought we had that in place. I have some suspicions but that would be hard to verify without core or repro. I can do loop test run to see if I can make it happen locally.

Crash and hang dump support across the entire supported OS matrix (except mobile) will be enabled with #39923.

[danmoseley]

@danmosemsft @maryamariyan see the bot's message above. The @dotnet/ncl comment is not highlighted as a mention

This is just a github rendering bug

[wfurt]

I could not reproduce it so far. This may need to waiter core or reocuerence.

[karelz]

Triage: We need core dump to investigate more ...

[karelz]

Does not seem to be actionable - unable to reproduce, no core dump for investigation ... closing for now.

[MihaZupan]

Failed in #51826, looks like there is a dump available

[karelz]

Note: I copied the dump locally in case it disappears before we get to it ...

[ManickaP]

Triage: punting.

[wfurt]

failed again in https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-61592-merge-a58e4d66b85d4d64af/System.Net.Security.Tests/1/console.1aa05afd.log?sv=2019-07-07&se=2021-12-05T13%3A40%3A08Z&sr=c&sp=rl&sig=jMs9%2BnoRb5nvfChlrWnIlpTTlE%2FyexQpyxNXwl1qoLQ%3D

Process terminated. Assertion failed.
Expected 164 <= 0
   at System.Net.ArrayBuffer.Discard(Int32 byteCount) in /_/src/libraries/Common/src/System/Net/ArrayBuffer.cs:line 67
   at System.Net.SafeDeleteSslContext.ReadPendingWrites() in /_/src/libraries/System.Net.Security/src/System/Net/Security/Pal.OSX/SafeDeleteSslContext.cs:line 259
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.OSX.cs:line 263
   at System.Net.Security.SslStreamPal.InitializeSecurityContext(SafeFreeCredentials& credential, SafeDeleteSslContext& context, String targetName, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStreamPal.OSX.cs:line 52
   at System.Net.Security.SecureChannel.GenerateToken(ReadOnlySpan`1 inputBuffer, Byte[]& output) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 803
   at System.Net.Security.SecureChannel.NextMessage(ReadOnlySpan`1 incomingBuffer) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SecureChannel.cs:line 725
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 352
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs:line 38
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Security.SslStream.ProcessAuthenticationAsync(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 202
   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 296
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 282
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.cs:line 263
   at System.Net.Security.Tests.SslStreamSniTest.<>c__DisplayClass1_1.<SslStream_ServerCallbackAndLocalCertificateSelectionSet_Throws>b__2() in /_/src/libraries/System.Net.Security/tests/FunctionalTests/SslStreamSniTest.cs:line 78

it seems like SslStream_ServerCallbackAndLocalCertificateSelectionSet_Throws test again. We can perhaps disable it if this becomes too bothersome. But it feels like we need better synchronization for Disposing.

[hoyosjs]

This time in Linux x64

https://helix.dot.net/api/2019-06-17/jobs/84a6a55e-cf80-4aca-b0a5-fc4afb645eea/workitems/System.Net.Security.Tests/console

[wfurt]

The linked failure is different as it does not live in macOS PAL. It is tracked by #66818.