Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AF: System.Net.Security.SslStream.ConsumeBufferedBytes (Linux/OSX x64 Debug) #20507

Closed
jkotas opened this issue Jan 31, 2020 · 6 comments · Fixed by #32267
Closed

AF: System.Net.Security.SslStream.ConsumeBufferedBytes (Linux/OSX x64 Debug) #20507

jkotas opened this issue Jan 31, 2020 · 6 comments · Fixed by #32267
Assignees
@wfurt
Labels
area-System.Net blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' bug
Milestone
5.0.0

Comments

@jkotas
Copy link
Member

jkotas commented Jan 31, 2020

Failed in #14381:

  Discovering: System.Net.Security.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Security.Tests (found 170 of 277 test cases)
  Starting:    System.Net.Security.Tests (parallel test collections = on, max threads = 2)
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_SelfSignedClientEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ServerEKUClientAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ClientEKUServerAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUServerAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
CLR: Managed code called FailFast, saying "   at System.Diagnostics.DebugProvider.Fail(String message, String detailMessage)
   at System.Diagnostics.Debug.Fail(String message, String detailMessage)
   at System.Diagnostics.Debug.Assert(Boolean condition, String message, String detailMessage)
   at System.Diagnostics.Debug.Assert(Boolean condition)
   at System.Net.Security.SslStream.ConsumeBufferedBytes(Int32 byteCount)
   at System.Net.Security.SslStream.ReceiveBlobAsync(SslReadAsync adapter, Byte[] buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Net.Security.ProtocolToken, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Security.SslStream.<ReceiveBlobAsync>d__189, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Net.Security.ProtocolToken, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Security.SslStream.<ReceiveBlobAsync>d__189, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Net.Security.ProtocolToken, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Security.SslStream.<ReceiveBlobAsync>d__189, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32 result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(Task`1 task, Int32 result)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Int32 result)
   at System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|205_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|205_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|205_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|205_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ThreadPoolGlobals.<>c.<.cctor>b__4_0(Object state)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _)
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags)
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError)
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Byte[] socketAddress, Int32 socketAddressSize, SocketFlags receivedFlags, SocketError socketError)
   at System.Net.Sockets.SocketAsyncContext.BufferMemoryReceiveOperation.InvokeCallback(Boolean allowPooling)
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1[[System.Net.Sockets.SocketAsyncContext.ReadOperation, System.Net.Sockets, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ProcessAsyncOperation(ReadOperation op)
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op)
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()"

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

Unhandled Exception:
System.Diagnostics.DebugProvider+DebugAssertException:    at System.Diagnostics.DebugProvider.Fail(String message, String detailMessage)
   at System.Diagnostics.Debug.Fail(String message, String detailMessage)
   at System.Diagnostics.Debug.Assert(Boolean condition, String message, String detailMessage)
   at System.Diagnostics.Debug.Assert(Boolean condition)
   at System.Net.Security.SslStream.ConsumeBufferedBytes(Int32 byteCount)
   at System.Net.Security.SslStream.ReceiveBlobAsync(SslReadAsync adapter, Byte[] buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Net.Security.ProtocolToken, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Security.SslStream.<ReceiveBlobAsync>d__189, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Net.Security.ProtocolToken, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Security.SslStream.<ReceiveBlobAsync>d__189, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Net.Security.ProtocolToken, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Security.SslStream.<ReceiveBlobAsync>d__189, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32 result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(Task`1 task, Int32 result)
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Int32 result)
   at System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|205_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|205_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(Object s)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|205_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(Thread threadPoolThread)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|205_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ThreadPoolGlobals.<>c.<.cctor>b__4_0(Object state)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _)
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags)
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError)
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Byte[] socketAddress, Int32 socketAddressSize, SocketFlags receivedFlags, SocketError socketError)
   at System.Net.Sockets.SocketAsyncContext.BufferMemoryReceiveOperation.InvokeCallback(Boolean allowPooling)
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1[[System.Net.Sockets.SocketAsyncContext.ReadOperation, System.Net.Sockets, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ProcessAsyncOperation(ReadOperation op)
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op)
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

=================================================================
	Native stacktrace:
=================================================================
	0x7f40b62c4bad - /root/helix/work/correlation/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.so : 
	0x7f40b626aeb5 - /root/helix/work/correlation/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.so : 
	0x7f40b62c4241 - /root/helix/work/correlation/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.so : 
	0x7f40b7149e70 - /lib64/libpthread.so.0 : 
	0x7f40b6c6c57f - /lib64/libc.so.6 : gsignal
	0x7f40b6c56895 - /lib64/libc.so.6 : abort
	0x7f40b635512f - /root/helix/work/correlation/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.so : 
	0x7f40b63579d4 - /root/helix/work/correlation/shared/Microsoft.NETCore.App/5.0.0/libcoreclr.so : 
	0x42143715 - Unknown

=================================================================
	Telemetry Dumper:
=================================================================
Could not exec mono-hang-watchdog, expected on path '/usr/local/etc/../bin/mono-hang-watchdog' (errno 2)
Pkilling 0x139915920865088x from 0x139915302680320x
Pkilling 0x139915300579072x from 0x139915302680320x
Pkilling 0x139915328808704x from 0x139915302680320x
Pkilling 0x139915817838336x from 0x139915302680320x
Stack overflow in unmanaged: IP: 0x7f40b7175d6d, fault addr: 0x7f40b1199fe8
Stack overflow: IP: 0x7f40b717aee7, fault addr: 0x7f40b1190ff8
Stacktrace:
domain required for stack walk/root/helix/work/workitem
----- end Fri Jan 31 14:05:40 UTC 2020 ----- exit code 1 ----------------------------------------------------------
@maryamariyan maryamariyan added the untriaged New issue has not been triaged by the area owner label Feb 3, 2020
@jkotas
Copy link
Member Author

jkotas commented Feb 4, 2020 •
edited
Loading

Hit in #31705 on CoreCLR. This is not a Mono specific crash.

@jkotas jkotas added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Feb 4, 2020
@jkotas jkotas changed the title AF: System.Net.Security.SslStream.ConsumeBufferedBytes (mono Linux x64 Debug) AF: System.Net.Security.SslStream.ConsumeBufferedBytes (Linux/OSX x64 Debug) Feb 4, 2020
@stephentoub
Copy link
Member

@wfurt, we're hitting this a bunch. Can you take a look?

@stephentoub
Copy link
Member

Another case here:
https://dev.azure.com/dnceng/public/_build/results?buildId=513117&view=ms.vss-test-web.build-test-results-tab&runId=16262410&resultId=176999&paneView=attachments

  Discovering: System.Net.Security.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Security.Tests (found 177 of 277 test cases)
  Starting:    System.Net.Security.Tests (parallel test collections = on, max threads = 2)
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_SelfSignedClientEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ServerEKUClientAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ClientEKUServerAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUServerAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
Process terminated. Assertion failed.
   at System.Net.Security.SslStream.ConsumeBufferedBytes(Int32 byteCount) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1080
   at System.Net.Security.SslStream.ReceiveBlobAsync(SslReadAsync adapter, Byte[] buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 526
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 329
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 785
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3242
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 398
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncValueTaskMethodBuilderT.cs:line 66
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|205_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1034
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 329
   at System.Threading.ThreadPoolGlobals.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 41
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1049
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 869
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 776
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 407
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Byte[] socketAddress, Int32 socketAddressSize, SocketFlags receivedFlags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 122
   at System.Net.Sockets.SocketAsyncContext.BufferMemoryReceiveOperation.InvokeCallback(Boolean allowPooling) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 490
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 888
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 1295
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 332
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 659

@stephentoub stephentoub added bug and removed untriaged New issue has not been triaged by the area owner labels Feb 8, 2020
@stephentoub stephentoub added this to the 5.0 milestone Feb 8, 2020
@wfurt
Copy link
Member

wfurt commented Feb 9, 2020

Can you please take another look at #1949 @stephentoub ?
With that, _internalBuffer and ConsumeBufferedBytes() are no longer used for the handshake.
I have a suspicion of what is going on but that PR fixes few other regressions caused by #453.

@steveharter
Copy link
Member

Another from #32184:

System.Net.Security.Tests
  Discovering: System.Net.Security.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Security.Tests (found 170 of 277 test cases)
  Starting:    System.Net.Security.Tests (parallel test collections = on, max threads = 2)
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_SelfSignedClientEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ServerEKUClientAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ClientEKUServerAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUServerAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
CLR: Managed code called FailFast, saying "Header buffer is not allocated.
SslStream#557235912.DetectFraming
   at System.Diagnostics.DebugProvider.Fail(String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/DebugProvider.cs:line 20
   at System.Diagnostics.Debug.Fail(String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 119
   at System.Net.NetEventSource.Fail(Object thisOrContextObject, Object message, String memberName) in /_/src/libraries/Common/src/System/Net/Logging/NetEventSource.Common.cs:line 287
   at System.Net.Security.SslStream.DetectFraming(Byte[] bytes, Int32 length) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 1177
   at System.Net.Security.SslStream.ReceiveBlobAsync(SslReadAsync adapter, Byte[] buffer, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 471
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Net.Security.ProtocolToken, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Security.SslStream.<ReceiveBlobAsync>d__186, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(Object s) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 290
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Net.Security.ProtocolToken, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Security.SslStream.<ReceiveBlobAsync>d__186, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 329
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Net.Security.ProtocolToken, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Net.Security.SslStream.<ReceiveBlobAsync>d__186, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext() in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 307
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 785
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3242
   at System.Threading.Tasks.Task.FinishContinuations() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3214
   at System.Threading.Tasks.Task`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].TrySetResult(Int32 result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 398
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(Task`1 task, Int32 result) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 441
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetResult(Int32 result) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncValueTaskMethodBuilderT.cs:line 66
   at System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|202_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext() in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 991
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|202_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionContextCallback(Object s) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 290
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|202_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 329
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream.<<FillBufferAsync>g__InternalFillBufferAsync|202_0>d`1[[System.Net.Security.SslStream.SslReadAsync, System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext() in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 307
   at System.Threading.ThreadPoolGlobals.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 41
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1049
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 869
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 776
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 407
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Byte[] socketAddress, Int32 socketAddressSize, SocketFlags receivedFlags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 122
   at System.Net.Sockets.SocketAsyncContext.BufferMemoryReceiveOperation.InvokeCallback(Boolean allowPooling) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 490
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1[[System.Net.Sockets.SocketAsyncContext.ReadOperation, System.Net.Sockets, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ProcessAsyncOperation(ReadOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 888
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 1295
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 332
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 664
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 40
   at System.Threading.Thread.StartCallback() in /_/src/mono/netcore/System.Private.CoreLib/src/System/Threading/Thread.Mono.cs:line 259"

@wfurt wfurt mentioned this issue Feb 13, 2020
Merged
@eerhardt
Copy link
Member

Another here:

https://helix.dot.net/api/2019-06-17/jobs/cc457fa9-1d5a-44b2-81c6-8bc9b987594a/workitems/System.Net.Security.Tests/console

https://dev.azure.com/dnceng/public/_build/results?buildId=521493&view=logs&jobId=71afe768-e493-597c-228c-037834193aeb&j=71afe768-e493-597c-228c-037834193aeb&t=3e74a8b7-5e62-58f8-6a85-9e36ef39d1c1

Process terminated. Assertion failed.
   at System.Net.Security.SslStream.ConsumeBufferedBytes(Int32 byteCount) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 881
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 337
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 290
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 329
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 307
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 785
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3242
   at System.Threading.Tasks.Task.FinishContinuations() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 3214
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs:line 398
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1 task, TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 441
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult result) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncValueTaskMethodBuilderT.cs:line 66
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|192_0[TIOAdapter](TIOAdapter adap, ValueTask`1 task, Int32 min, Int32 initial) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs:line 835
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 290
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 172
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 329
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() in /_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs:line 307
   at System.Threading.ThreadPoolGlobals.<>c.<.cctor>b__4_0(Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 41
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.InvokeContinuation(Action`1 continuation, Object state, Boolean forceAsync, Boolean requiresExecutionContextFlow) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 1049
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.OnCompleted(SocketAsyncEventArgs _) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.Tasks.cs:line 869
   at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.cs:line 776
   at System.Net.Sockets.SocketAsyncEventArgs.CompletionCallback(Int32 bytesTransferred, SocketFlags flags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 407
   at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Byte[] socketAddress, Int32 socketAddressSize, SocketFlags receivedFlags, SocketError socketError) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs:line 122
   at System.Net.Sockets.SocketAsyncContext.BufferMemoryReceiveOperation.InvokeCallback(Boolean allowPooling) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 490
   at System.Net.Sockets.SocketAsyncContext.OperationQueue`1.ProcessAsyncOperation(TOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 888
   at System.Net.Sockets.SocketAsyncContext.ProcessAsyncReadOperation(ReadOperation op) in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 1295
   at System.Net.Sockets.SocketAsyncContext.ReadOperation.System.Threading.IThreadPoolWorkItem.Execute() in /_/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncContext.Unix.cs:line 332
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs:line 659
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() in /_/src/coreclr/src/System.Private.CoreLib/src/System/Threading/ThreadPool.CoreCLR.cs:line 29
./RunTests.sh: line 161: 13137 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Security.Tests.runtimeconfig.json --depsfile

@ghost ghost locked as resolved and limited conversation to collaborators Dec 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@wfurt wfurt

Labels
area-System.Net blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' bug
Projects
None yet
Milestone
5.0.0
Development

Successfully merging a pull request may close this issue.

fix buffer handling in Tls handshake wfurt/runtime
6 participants
@stephentoub @maryamariyan @jkotas @steveharter @eerhardt @wfurt