Operation could destabilize the runtime

[NikitaEgorov]

• NUnit Adapter 3.10.0.21
• TFS 16.131.27701.1
• Code coverage enabled = true

NUnit Adapter 3.10.0.21: Test execution started
2018-06-18T12:18:41.2125333Z Running all tests in E:\Agent\_work\209\s\src\*****.Tests.Unit.dll
2018-06-18T12:18:42.8180637Z Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in E:\Agent\_work\209\s\src\**************.Tests.Unit.dll
2018-06-18T12:18:42.8180637Z The NUnit 3 driver encountered an error while executing reflected code.
2018-06-18T12:18:42.8620107Z    at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
2018-06-18T12:18:42.8620107Z    at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
2018-06-18T12:18:42.8620107Z    at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
2018-06-18T12:18:42.8629873Z    at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
2018-06-18T12:18:42.8629873Z    at NUnit.Engine.Runners.TestDomainRunner.LoadPackage()
2018-06-18T12:18:42.8629873Z    at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
2018-06-18T12:18:42.8629873Z    at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter)
2018-06-18T12:18:42.8629873Z    at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)
2018-06-18T12:18:42.8629873Z    at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, TestFilter filter) in D:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 229
2018-06-18T12:18:42.8629873Z Innerexception: System.Security.VerificationException: Operation could destabilize the runtime.
2018-06-18T12:18:42.8629873Z    at NUnit.Framework.Api.FrameworkController..ctor(String assemblyNameOrPath, String idPrefix, IDictionary settings) in C:\src\nunit\nunit\src\NUnitFramework\framework\Api\FrameworkController.cs:line 68

[OsirisTerje]

Can you please upload a repro project?
Make it as small as you can, it should compile "as is".

[NikitaEgorov]

It is internal project

[siprbaum]

Perhaps try reducing the internal project, e.g. remove code so long as it still reproduces.
If you have a minimal reproducer with your production code, the next step would be to "rewrite" it
so you can publish it here. Otherwise I'd guess it is really hard to analyse the issue without further information.

[OsirisTerje]

We need some more information on this, the normal way is that you provide a similar working repro project. We dont need to see anything from your internal project, but we need to know what you are trying to do.
Running a normal test project with VSTS (which is ahead of TFS dev 16) works as it should, using the ordinary VSTest task.

(Thanks @siprbaum , you beat me to it :-) )

[NikitaEgorov]

I reproduce it use empty nunit project

packages.config

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="NUnit" version="3.10.1" targetFramework="net472" />
  <package id="NUnit3TestAdapter" version="3.10.0" targetFramework="net472" />
</packages>

UnitTest1.cs

using System;

using NUnit.Framework;

namespace UnitTestProject1
{
    [TestFixture]
    public class UnitTest1
    {
        [Test]
        public void TestMethod1()
        {
            Console.WriteLine("test");
        }
    }
}

b7b3e470-73cd-11e8-ba0f-2dd83a1f0155.runsettings

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Include>
                <ModulePath>.*.dll$</ModulePath>
                <ModulePath>.*.exe$</ModulePath>
              </Include>
              <Exclude>
                <ModulePath>.*CPPUnitTestFramework.*</ModulePath>
              </Exclude>
            </ModulePaths>
            <Functions>
              <Exclude>
                <Function>^Fabrikam.UnitTest..*</Function>
                <Function>^std::.*</Function>
                <Function>^ATL::.*</Function>
                <Function>.*::__GetTestMethodInfo.*</Function>
                <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
                <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
              </Exclude>
            </Functions>
            <Attributes>
              <Exclude>
                <Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute>
                <Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute>
                <Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute>
                <Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute>
                <Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute>
              </Exclude>
            </Attributes>
            <Sources>
              <Exclude>
                <Source>.*\atlmfc\.*</Source>
                <Source>.*\vctools\.*</Source>
                <Source>.*\public\sdk\.*</Source>
                <Source>.*\microsoft sdks\.*</Source>
                <Source>.*\vc\include\.*</Source>
              </Exclude>
            </Sources>
            <CompanyNames>
              <Exclude>
                <CompanyName>.*microsoft.*</CompanyName>
              </Exclude>
            </CompanyNames>
            <PublicKeyTokens>
              <Exclude>
                <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken>
                <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken>
                <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken>
                <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken>
                <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken>
                <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken>
                <PublicKeyToken>^E361AF139669C375$</PublicKeyToken>
              </Exclude>
            </PublicKeyTokens>
            <UseVerifiableInstrumentation>False</UseVerifiableInstrumentation>
            <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
            <CollectFromChildProcesses>True</CollectFromChildProcesses>
            <CollectAspDotNet>False</CollectAspDotNet>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

run.bat

e:\15.8.0-preview-20180610-02\x64\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "UnitTestProject1.dll" /logger:"trx" /Settings:"b7b3e470-73cd-11e8-ba0f-2dd83a1f0155.runsettings" /TestAdapterPath:"e:\Test\Debug\"

[NikitaEgorov]

Microsoft (R) Test Execution Command Line Tool Version 15.8.0-preview-20180610-02
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
NUnit Adapter 3.10.0.21: Test execution started
Running all tests in e:\Test\Debug\UnitTestProject1.dll
Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in e:\Test\Debug\UnitTestProject1.dll
The NUnit 3 driver encountered an error while executing reflected code.
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
   at NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary`2 settings)
   at NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage)
   at NUnit.Engine.Runners.DirectTestRunner.LoadPackage()
   at NUnit.Engine.Runners.TestDomainRunner.LoadPackage()
   at NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded()
   at NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter)
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, TestFilter filter) in D:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 229
Innerexception: System.Security.VerificationException: Operation could destabilize the runtime.
   at NUnit.Framework.Api.FrameworkController..ctor(String assemblyNameOrPath, String idPrefix, IDictionary settings) in C:\src\nunit\nunit\src\NUnitFramework\framework\Api\FrameworkController.cs:line 68
NUnit Adapter 3.10.0.21: Test execution complete
No test is available in e:\Test\Debug\UnitTestProject1.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

[OsirisTerje]

1. Can you zip together the project and upload that, the code above should work.
2. Does it also fail when you don't use the runsettings file?
3. Does it also fail when you change the .net framework target to e.g. 4.7.1 or 4.6.1 ?
4. Does it also fail within Visual Studio ?

[OsirisTerje]

Ok, I can confirm it now. It is caused by the runsettings file.
If you change your runsettings file to use InProcDataCollectors instead it will work.
This needs to be properly documented. I'll add a seperate issue for that.

[NikitaEgorov]

file b7b3e470-73cd-11e8-ba0f-2dd83a1f0155.runsettings is generated by VSTest task
I can not change it manual

[OsirisTerje]

Yes, you can change it:

(MS also has noted the current behaviour as a bug, see links from @pvlakshm above. )

[ShreyasRmsft]

@OsirisTerje

The UseVerifiableInstrumentation is set to false by the VsTest task when the platform is acquired via the vstestplatform tool installer (gets the nuget package) as the profiler may not be available in the GAC and needs to be resolved using certain environment variables set by the VsTest task.

This is a must for code coverage to work when there is no VS installed on the target machine.

This will have to be fixed otherwise code coverage will not work for NUnit tests when run using the VsTest Platform Nuget package (No VS).

[OsirisTerje]

Thanks @ShreyasRmsft

With "have to be fixed", do you mean in the VSTest task or will the project need its own runsettings file?

[NikitaEgorov]

I added custom runsettings with True </ UseVerifiableInstrumentation>
Nothing changes

[NikitaEgorov]

All works fine with MSTest :(

[ShreyasRmsft]

@OsirisTerje is the NUnit adapter throwing this exception? Is the adapter reading the data collector node in the runsettings?

[OsirisTerje]

@ShreyasRmsft It is an exception coming from the NUnit Engine, so it is not the adapter itself, but the embedded NUnit engine. I am unsure about how this collector is intended to work, one of you guys did these changes earlier, and it seems that something here causes the engine to crash.
However, when I changed it to InProcDatacollector my repro worked.

And yes, the adapter reads the node and then sets some other properties:

@NikitaEgorov Did you change the runsettings as I described above? (not only setting the flag to true, but replacing the DataCollector with InProcDataCollector?

[ShreyasRmsft]

@OsirisTerje changing it to inproc is not the solution. Ideally in this case the engine should not be reading these settings at all as they are specific to code coverage.

[OsirisTerje]

@ShreyasRmsft The engine don't read MS properties, only NUnit properties. The adapter does the conversion as shown above. Anything else being done must happen on the datacollector side. And this code above is from MS, so I assume you are setting whatever is needed correctly here ?
The exception surfacing in the Engine doesn't seem to have anything to do with the Engine itself, but with whatever it is trying to load:

[ShreyasRmsft]

@OsirisTerje I do not have a lot of context on the change that was done.

But yes if the change that was made by one of us is trying to read this settings then it needs to be fixed. I haven't had time yet to see where exactly in the code this is happening. But Nunit should ideally ignore anything present in the code coverage node. Will get back to you on the fix.

[OsirisTerje]

@ShreyasRmsft I believe the code involved were written by @navin22, i think this PR #391 was the essential one. I also see a later modification by @drognanar to "Use InProcDataCollector setting to enable sequential run", PR #510, but this is not released yet, it's for 3.11.

[MoiMoiDG]

Hi there,

I have been told that this issue affects another issue (which I submitted).
I am quite new to this repo, so I cannot get what is the current status. Is anyone working on a solution to this issue?

Thanks!

[OsirisTerje]

@MoiMoiDG MS themselves are working on it on their side, but I'll check up on it.

[MoiMoiDG]

@OsirisTerje: Thanks! Is there any update on this?

[OsirisTerje]

@ShreyasRmsft Can you mail me on what I need to do with the adapter to have this fixed? (terje at hermit.no)

[OsirisTerje]

@ShreyasRmsft

Since we use sourcelink ( and many more do the same), using the pdb’s as a indicator for the “dll’s under test” is likely to cause more such issues. And the test framework dll's should not in any case be part of the profiling.

Do you call the profiler before or after you call the adapter?
Can we add the nunit dll’s to the exclude filter in the adapter ?

[ThomasPiskol]

Is there any update on this issue?

[mayankbansal018]

Sorry for such delay on this.

Issue Explanation:
When we instrument a dll which contains methods/classes marked with certain Security Attributes e.g. SecurityCritical or few more, then during code coverage these methods are likely to throw exception, since these attributes prohibit certain scenarios, e.g run-time modification of existing code, or writing to raw memory, etc.

Solutions:
We do have fix for such scenarios, where we delegate responsibility of writing to raw memory, to another dll, which we call Shim dll, which gets installed in GAC when VS is present, however in XCopyable scenario this fails.

We are planning to include Shim dll as part of our XCopy package, & also we plan to release code coverage nugets, which will allow users to take explicit dependency on Shim dll. I'll update the thread once we have done either.

For now only solution available to user would be use code coverage runsettings which will exclude NUnit dlls from getting instrumented.

[MoiMoiDG]

Hi @mayankbansal018,

Thanks for explaining the problem!

For now only solution available to user would be use code coverage runsettings which will exclude NUnit dlls from getting instrumented.

Could you please provide an example of such runsettings file where it works? I know it is a workaround, but it would be helpful.

Thanks!

[mayankbansal018]

@MoiMoiDG the following runsettings should help you

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <RunSettings> <DataCollectionRunSettings> <DataCollectors> <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"> <Configuration> <CodeCoverage> <ModulePaths> <Include> <ModulePath>.*.dll$</ModulePath> <ModulePath>.*.exe$</ModulePath> </Include> <Exclude> <ModulePath>.*CPPUnitTestFramework.*</ModulePath> <ModulePath>.*nunit.*</ModulePath> </Exclude> </ModulePaths> <Functions> <Exclude> <Function>^Fabrikam.UnitTest..*</Function> <Function>^std::.*</Function> <Function>^ATL::.*</Function> <Function>.*::__GetTestMethodInfo.*</Function> <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function> <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function> </Exclude> </Functions> <Attributes> <Exclude> <Attribute>^System.Diagnostics.DebuggerHiddenAttribute$</Attribute> <Attribute>^System.Diagnostics.DebuggerNonUserCodeAttribute$</Attribute> <Attribute>^System.Runtime.CompilerServices.CompilerGeneratedAttribute$</Attribute> <Attribute>^System.CodeDom.Compiler.GeneratedCodeAttribute$</Attribute> <Attribute>^System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverageAttribute$</Attribute> </Exclude> </Attributes> <Sources> <Exclude> <Source>.*\atlmfc\.*</Source> <Source>.*\vctools\.*</Source> <Source>.*\public\sdk\.*</Source> <Source>.*\microsoft sdks\.*</Source> <Source>.*\vc\include\.*</Source> </Exclude> </Sources> <CompanyNames> <Exclude> <CompanyName>.*microsoft.*</CompanyName> </Exclude> </CompanyNames> <PublicKeyTokens> <Exclude> <PublicKeyToken>^B77A5C561934E089$</PublicKeyToken> <PublicKeyToken>^B03F5F7F11D50A3A$</PublicKeyToken> <PublicKeyToken>^31BF3856AD364E35$</PublicKeyToken> <PublicKeyToken>^89845DCD8080CC91$</PublicKeyToken> <PublicKeyToken>^71E9BCE111E9429C$</PublicKeyToken> <PublicKeyToken>^8F50407C4E9E73B6$</PublicKeyToken> <PublicKeyToken>^E361AF139669C375$</PublicKeyToken> </Exclude> </PublicKeyTokens> <UseVerifiableInstrumentation>False</UseVerifiableInstrumentation> <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses> <CollectFromChildProcesses>True</CollectFromChildProcesses> <CollectAspDotNet>False</CollectAspDotNet> </CodeCoverage> </Configuration> </DataCollector> </DataCollectors> </DataCollectionRunSettings> </RunSettings>

[MoiMoiDG]

Hi @mayankbansal018,

Thank you!

However, I still get the error in attachment. I attached the full solution too, for your convenience (it is just a small sample) ... :(
7_VsTest - testAssemblies.zip
ESXD-NUTests.zip

[OsirisTerje]

@MoiMoiDG @mayankbansal018 : The project works when in Visual Studio, using the runsettings file, and failed without. After a bit hazzling it started working in both cases. Not really sure why. But in your log it says the runsettings ARE loaded, and I see you still get the same error.
@mayankbansal018 : This is a very annoying issue, and I there seems to be some caching issues involved. The issue is not stable.

[mayankbansal018]

@OsirisTerje the caching of file issue can be only with Visual Studio, though we have not seen it too often, adding @AbhitejJohn, @ManishJayaswal to see if they have more info.

@MoiMoiDG can you confirm if this ever fails for you in CLI when used with above runsettings?

[MoiMoiDG]

Hi everyone,

Thank you for the support!

@mayankbansal018: I am not sure that I understood your question. The log comes from a VM attached to TFS 2018.3 and "Test platform version = Installed by Tools Installer".

[samneefs]

The following seems to be related: microsoft/vstest#1950
I had the error after switching to Newtonsoft.Json version 12. Turning off code coverage or using another json serializer are workarounds.

[mayankbansal018]

Fixed with microsoft/vstest#1997, we have also released latest nuget package (16.0.2 preview) of test platform with above change, please upgrade to it.

[OsirisTerje]

@mayankbansal018 Do you refer to the VSTest task or to which nuget package?

[mayankbansal018]

@OsirisTerje I'm referring to nuget that gets installed via Test Platform nuget installer task in pipeline. If users facing this issue upgrade to latest preview, then it should be resolved for them

[sjblack]

The latest version of the console (16.0.2 preview) did not resolve the problem for me.

[DeadlyDragon]

Is there an update on this? Running 16.2.0 and still hitting this issue. We need code coverage from the nunit test results in order to achieve our targets.

[ShreyasRmsft]

Please take a look at the comments in microsoft/vstest#1997. I have provided details on the fix. Make sure you have followed all the instructions and even after that you are facing issues please provide the full set of debug logs for the pipeline run and we will take a look.

[samneefs]

The following seems to be related: microsoft/vstest#1950
I had the error after switching to Newtonsoft.Json version 12. Turning off code coverage or using another json serializer are workarounds.

Seems to be fixed with the combination of:
Latest Newtonsoft.Json 12.0.2 (doesn't work with 12.0.1)
Latest stable Test platform: 16.2 (should be >16.0.2)
Latest VSTest task 2.156.2 (should be >2.153.0)