TaskScheduler UnobservedTaskException on long run of WCF 4.7.0 on .NetCore 3.1 (Win)

[SergeAgeyev]

Describe the bug
I our application (it uses a lot of WCF services with NetTcpBinding and HttpBinding), we observe TaskScheduler UnobservedTaskException after ~20 hours of operation when using WCF 4.7.0 on NetCore 3.1 (on Windows). The app using WCF to connect to Onvif video cameras and other WCF services as well.
By couple of reasons, we have to terminate application on UnobservedTaskException as system may become unstable in that case, so this prevent app to function properly.
The same code under full .Net Framework works fine.

To Reproduce
Run WCF-intense code for 24+ hours, then in some cases you see UnobservedTaskException
Unfortunately, there is no more specific information at the moment.
Probably this may somehow be related to network communication issues as well.

Expected behavior
The system should operate in a stable way.

Additional context

2020-06-22 06:00:46,895 [2] ERROR TaskScheduler_UnobservedTaskException Unhandled Exception

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The given key 'System.ServiceModel.Channels.RequestReplyCorrelator+Key: {urn:uuid:0828ae28-df71-4611-aeb6-XXXXXXXX, System.ServiceModel.Dispatcher.DuplexChannelBinder+IDuplexRequest}' was not present in the dictionary.)

---> System.Collections.Generic.KeyNotFoundException: The given key 'System.ServiceModel.Channels.RequestReplyCorrelator+Key: {urn:uuid:0828ae28-df71-4611-aeb6-f6a0bbdbc929, System.ServiceModel.Dispatcher.DuplexChannelBinder+IDuplexRequest}' was not present in the dictionary.

   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)

   at System.ServiceModel.Channels.RequestReplyCorrelator.System.ServiceModel.Channels.IRequestReplyCorrelator.Find[T](Message reply, Boolean remove)

   at System.ServiceModel.Dispatcher.DuplexChannelBinder.HandleRequestAsReplyCore(Message message)

   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequestAsReply(RequestContext request)

   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)

   at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)

   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)

   at System.Runtime.TaskHelpers.<>c__1`1.<ToApm>b__1_1(Task`1 antecedent, Object obj)

   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)

--- End of stack trace from previous location where exception was thrown ---

   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)

   --- End of inner exception stack trace ---

2020-06-22 06:00:46,897 [2] ERROR TaskScheduler_UnobservedTaskException Unhandled Exception

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The given key 'System.ServiceModel.Channels.RequestReplyCorrelator+Key: {urn:uuid:49a1f53f-24ea-4ad5-b319-YYYYYYYY, System.ServiceModel.Dispatcher.DuplexChannelBinder+IDuplexRequest}' was not present in the dictionary.)

---> System.Collections.Generic.KeyNotFoundException: The given key 'System.ServiceModel.Channels.RequestReplyCorrelator+Key: {urn:uuid:49a1f53f-24ea-4ad5-b319-b14401fe9630, System.ServiceModel.Dispatcher.DuplexChannelBinder+IDuplexRequest}' was not present in the dictionary.

   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)

   at System.ServiceModel.Channels.RequestReplyCorrelator.System.ServiceModel.Channels.IRequestReplyCorrelator.Find[T](Message reply, Boolean remove)

   at System.ServiceModel.Dispatcher.DuplexChannelBinder.HandleRequestAsReplyCore(Message message)

   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequestAsReply(RequestContext request)

   at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)

   at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)

   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)

   at System.Runtime.TaskHelpers.<>c__1`1.<ToApm>b__1_1(Task`1 antecedent, Object obj)

   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)

--- End of stack trace from previous location where exception was thrown ---

   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)

   --- End of inner exception stack trace ---

[HongGit]

@mconnew ?

[mconnew]

This was caused by switching from Hashtable to Dictionary and there being a change in behavior with the item getter. Dictionary[key] throws if the key isn't in the dictionary, whereas Hashtable[key] returns null. The scenario that this will occur under is when a callback happens and the callback method doesn't complete before something happens to the channel to shut it down. This would normally be a fault or abort as a regular close will wait for the request to be completed before continuing with the close. A close can timeout though, which turns it into an abort. I'm going to revert back to using Hashtable as we are only taking advantage of the generic type on the key, and it has some other concurrency advantages of being able to do lock free reads.