System.Security.VerificationException: Operation could destabilize the runtime

[lg2de]

Issue Description

I've updated the pipeline to use latest version of VSTest provided by "Visual Studio Test Platform Installer". Tests using "AutoMapper" now fail with VerificationException: Operation could destabilize the runtime.

There are several issues in github in relation to this issue.
It was reported in relation to NUnit (nunit/nunit3-vs-adapter#519) and AutoMapper (AutoMapper/AutoMapper#3254) too.
I tried to implement all the hint found on these and linked issue. The tests still fail.

I've running DevOps Server 2019 U1 with agent manually updated to 2.163.1.
I'm using always the latest vstest version (currently 16.4.0), so the bug microsoft/vstest#1997 should be fixed already.

I've created runsettings configuration file according to the documentation. There is activated the option UseVerifiableInstrumentation which should fix the Exception.

But the VSTest task overwrites the configuration.

Overriding UseVerifiableInstrumentation field to false in the runsettings file.

1. Why the task overrides my explicit setting? Similar question is not yet answered.
2. How to fix the problem?

Required Information

Question, Bug, or Feature? "Question" first, maybe (still) "Bug"

Enter Task Name: VSTest

Environment

• Server - OnPremise Version Dev17.M153.5
• Agent - private, hosted on Windows Server 2016, agent version 2.163.1

Task logs

logs_77777.zip

Error logs

==============================================================================
Task         : Visual Studio Test
Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
Version      : 2.153.9
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest
==============================================================================
[...]
Microsoft (R) Test Execution Command Line Tool Version 16.4.0
[...]
##[debug]Data collector already present, will not add the node.
Overriding UseVerifiableInstrumentation field to false in the runsettings file.
##[debug]Successfully added code coverage settings details to runsettings file.
[...]
  X NewData_MockedServiceProxy_NewDataOnProxyServiceGetCalled [755ms]
  Error Message:
   Test method AIS.EDA.Communication.Freeze2.Tests.Port.ClientProxyFacadeTests.NewData_MockedServiceProxy_NewDataOnProxyServiceGetCalled threw exception: 
System.TypeInitializationException: The type initializer for 'AIS.EDA.Communication.Freeze2.ConvertE134' threw an exception. ---> System.Security.VerificationException: Operation could destabilize the runtime.
  Stack Trace:
      at AutoMapper.MapperConfiguration..ctor(Action`1 configure) in C:\projects\automapper\src\AutoMapper\MapperConfiguration.cs:line 57
   at AIS.EDA.Communication.Freeze2.ConvertE134..cctor() in C:\build\_work\139\s\EDA\Source\Framework\AIS.EDA.Communication.Freeze2\ConvertE134.cs:line 21

[ShreyasRmsft]

In order to collect code coverage with the vstest platform tool installer UseVerifiableInstrumentation needs to be set to false.

This was true until testplatform version 16.1.x or greater was released. In versions of the test platform later than this UseVerifiableInstrumentation is no longer needed to be set to true to collect code coverage for certain nunit and xunit tests.

When OnPremise Version Dev17.M153.5 was released/forked off this version of the testplatform had not yet been released.

In later releases we modified the task behavior to stop overriding UseVerifiableInstrumentation to false if the testplatform version is greater than 16.1.x (we could only do this after the testplatform verison was released and we were aware it fixed this behavior).

As of now if you are running into this the workaround is to have VS installed and to not use the test platform installer. This has already been fixed in the next OnPrem release and should be out soon.

[lg2de]

Thanks for the explanation. That's really helpful!
For my complete understanding: The problem must be fixed (is fixed?) in the task implementation which is part of the DevOps Server software. It is not part of the agent software.
Correct?

Is there a release plan for an OnPremise version with the fix?

During further experimentation I identified another workaround: I have already enabled code coverage in the specified runsettings file. So I do not need to activate it with the VSTest task. If the option "Code coverage enabled" is deactivated the override is skipped and the tests are completed successfully.

[ShreyasRmsft]

@lg2de yeah that works too. Yes the fix will be there in AzureDevOps Server 2020