DiagnosticSource.dll not found exception in RoleEnvironment.Changed

[lmolkova]

Symptoms

When application is running in cloud service and subscribes to RoleEnvironment change events, it gets
SerializationException: Type is not resolved for member 'System.Diagnostics.Activity,System.Diagnostics.DiagnosticSource, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'

[SerializationException: Type is not resolved for member 'System.Diagnostics.Activity,System.Diagnostics.DiagnosticSource, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'.]
   System.AppDomain.get_Id() +0
   <CrtImplementationDetails>.DoCallBackInDefaultDomain(IntPtr function, Void* cookie) +126
   <CrtImplementationDetails>.LanguageSupport.InitializeDefaultAppDomain(LanguageSupport* ) +62
   <CrtImplementationDetails>.LanguageSupport._Initialize(LanguageSupport* ) +323
   <CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* ) +37
...

Assembly Binding Log Viewer/fuselogvw shows following issue:

*** Assembly Binder Log Entry  (6/6/2017 @ 3:06:07 PM) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable  C:\Program Files (x86)\IIS Express\iisexpress.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: DisplayName = System.Diagnostics.DiagnosticSource, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
 (Fully-specified)
LOG: Appbase = file:///C:/Program Files (x86)/IIS Express/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = iisexpress.exe
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: No application configuration file found.
LOG: Using host configuration file: C:\Users\lmolkova.EUROPE\Documents\IISExpress\config\aspnet.config
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Post-policy reference: System.Diagnostics.DiagnosticSource, Version=4.0.2.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/IIS Express/System.Diagnostics.DiagnosticSource.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/IIS Express/System.Diagnostics.DiagnosticSource/System.Diagnostics.DiagnosticSource.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/IIS Express/System.Diagnostics.DiagnosticSource.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/IIS Express/System.Diagnostics.DiagnosticSource/System.Diagnostics.DiagnosticSource.EXE.
LOG: All probing URLs attempted and failed.

Note that it is issexpress that could not load the DiagnosticSource assembly, so the presence of dll in the siteroot of the application does not help.

Root cause

• At the moment when Application_Start is called, there is an active Activity created to track incoming request. Activity declared in the DiagnosticSource.dll
• Activity.Current is stored in LogicalCallContext
• When RoleEnvironment.Changed is called, it causes cross AppDomain call and iis tries to deserialize activity from the LogicalCallContext.
• The causes iis to look for DiagnosticSource dll and it fails.

Workarounds

0. Install preview DiagnosticSource package with the fix

1. Put DiagnosticSource dll to one of the folders where iis looks for dlls ("%PROGRAMFILES(X86)%\IIS Express" - IIS Express; %systemroot%\System32\inetsr - IIS).

2. (Carefully). If you have .NET 4.6 is installed on the dev/test/prod machines, you can manually change reference to DiagnosticSource package to net46 one in the VS by removing reference and adding new one to the
   SolutionDir\packages\System.Diagnostics.DiagnosticSource.4.4.0-preview2-<latest>\lib\net46\System.Diagnostics.DiagnosticSource.dll or by changing csproj file directly

3. (Carefully) Before calling RoleEnvironment.Changed, clean up current activity with the following code: while (Activity.Current != null) { Activity.Current.Stop(); }
   As the downside, the first request in the instance may not be correlated with dependencies.
   Also, there may be other RoleEnvironment calls that cause cross AppDomain calls, so it may not be suitable in all cases.

[SergeyKanzhelev]

So this will repro for the Application Insights SDK for FW 4.0 as well? It looks like a design flaw of diagnostics source. Should we keep using CallContext at all?

[SergeyKanzhelev]

Or perhaps store the string instead of type in CallContext?

[SergeyKanzhelev]

Another question - is there anything that can tell "do not deserialize"? How HttpContext not breaking things?

[lmolkova]

Yes. it should repro on NET4.0 for OperationContextForCallContext.

It makes sense to store a string in the CallContext, however, it would not work well with Activity.

Another approach may be to avoid supporting Activity/OperationContextForCallContext outside of the appdomain by just using slot specific to appdomain. I would prefer this one.

[lmolkova]

Another question - is there anything that can tell "do not deserialize"?

I will check.

How HttpContext not breaking things?

HttpContext.Current is not in the logical call context. It's in synchronization context and never leaves AppDomain.

[pharring]

Make Activity derive from MarshalByRef to avoid serialization. Would be a breaking change, though.

[lmolkova]

@pharring
I'm looking into documentation and see following:

When a remote method call is made to an object in another AppDomain, the CallContext class generates a LogicalCallContext instance that travels along with the remote call. Only objects that expose the ILogicalThreadAffinative interface and are stored in the CallContext are propagated outside the AppDomain in a LogicalCallContext. Objects that do not support this interface are not transmitted in LogicalCallContext instances with remote method calls.

That seems to be wrong: Activity does not implement ILogicalThreadAffinative and therefore should not even be propagated. But it is, which is quite easy to reproduce.

There is no problem with serialization, the problem is that DiagnosticSource dll where Activity is defined is not loaded in IIS/ISSExpress, I'm not sure if MarshalByRefObject would help, I'll check.
I hope breaking change on NET4.5 only before the first stable release is still possible.

[pharring]

For locating the required assembly:
Do we have control over the "other" AppDomain in this scenario? i.e. do we own the code that created it? If so, it might be as simple as updating the new AppDomain's AppDomainSetup object with the necessary paths (copying it from the primary AppDomain). Can't remember whether that's the "AppBase" or the probing path, but it's probably one of those.

[lmolkova]

those AppDomains are controlled by IIS, so I doubt there is anything we can do, but I will check if there are some useful configuration options.

[lmolkova]

Glimpse had the same issue #632. There was a suggestion to wrap Activity with ObjectHandler. I've played with it and it does the trick.

[pharring]

Nice find. That'll work.

[AlexBulankou]

Fixed in stable version of Diagnostic Source

[ek68794998]

@AlexBulankou Are you referring to 4.4.0 (stable)? I am still receiving this exception in an Application Insights project using 4.4.0. It only occurs using net45 version of the DLL (forcing net46 seems to work around the issue).

[lmolkova]

@AlexBulankou
The fix was not included into the DiagnosticSource 4.4.0. It will be shipped with 4.4.1 or 4.5.0, whatever is next.

@ekumlin
Your observations are right. If you experience this issue, please check workarounds in the feature description. Let me know if they don't work for you

[ek68794998]

The net46 workaround works for me locally, but not in Azure. To mitigate the issue we are just catching the exception for now, though this obviously not the best way forward. However, this will work for us for now, and we will anxiously await the next stable release.

Thanks!

[TripleEmcoder]

I am seeing this error ("Type is not resolved for member") in a bit different scenario. I think in my case the correct solution would be to drop the CallContext before entering the second AppDomain. Do you know how to achieve that? I am guessing that I would need to create a Task to create a child context, and then clear the Activity in CallContext by name, and hope that when returning to the Task caller it will be restored. What do you think?

[NielsFilter]

Upgrading to the stable 4.4.1 didn't work for us.
Had to upgrade to the latest 4.5.0-preview

[lmolkova]

@TripleEmcoder this issue affects any cross AppDomain calls. I suggest that you use latest 4.5.0-preview of DiagnosticSource from dotnet myget feed: https://dotnet.myget.org/F/dotnet-core/api/v3/index.json

@NielsFilter you are right, 4.4.1 does NOT have a fix and it is shipped in 4.5.0