TFS Impact Tests are not working with NUnit tests

[ADBrinkman]

Issue description

When configuring the VS Test task to "Run only impacted Tests", code changes are not picked up by any of the affected NUnit tests. The tests are correctly discovered with the NUnit Adapter, but the TestSelector GetImpactedTests does not return any impacted tests.
New tests or failed tests are correctly picked up and run, just not tests for code changes.

If I use MSTest tests instead, these work as expected.

Initially I raised this issue on the Azure pipelines forum (see microsoft/azure-pipelines-tasks#9406 for full thread).
From this I understand TestImpact relies on the events to be in synchronous manner.

We tried running the VS Test task with a runsettings file specifying the CollectDataForEachTestSeparately which had the same problem:

<RunConfiguration>
<CollectDataForEachTestSeparately>true</CollectDataForEachTestSeparately>
</RunConfiguration>

It was suggested that CollectDataForEachTestSeparately does not work with OutOfProc data collectors, would you be able to confirm whether this is the case?
And if it is, is there a way to enable the events to be fired synchronously when using an OutOfProc Data Collector so that it will work with Test Impact Analysis?

Task logs

Full build debug logs are here:
Build logs - Impact tests not working.zip

Environment

• NUnit version 3.11.0
• NUnit Adapter version 3.12.0.0
• TFS is on-premise
• TFS Version 15.117.27414.0
• Visual Studio Enterprise 2017, version 15.9.5

[OsirisTerje]

@ADBrinkman I assume this can be related to #560 and possibly also #519 .
The CollectDataForEachTestSeparately will set the adapter to send two settings, Parallel = Off, and SynchronousEvents = true;

If you turn on NUnit Verbosity to full (= 5), you should see an info saying it is turned on, and the process will as an effect run slower.
If you see that, you also know that the two properties above has been set.

You can then also enable the Dump settings and see if they do come out in the right sequence during execution. What we have seen earlier was that it did, but there absolutely seems to be something wrong going on here, so we're very interested to see where this leads.
(See https://github.com/nunit/docs/wiki/Tips-And-Tricks for info on the dump settings)

[ADBrinkman]

Thanks, we have enabled the extra logging.
We can now see this message "CollectDataForEachTestSeparately is set, which is used to make InProcDataCollectors collect data for each test separately. No InProcDataCollectors can be found, thus the tests will run slower unnecessarily." is output in the build log.

The dump logs look ok to me, but it's not in the format shown in #560.
Dump logs are here:
NUnit Dump.zip

Build Log is here:
Build logs with NUnit logging enabled.zip

Please let me know if there is other information we should gather

[OsirisTerje]

@rprouse I believe it is NUnit itself that does this. We have other issues pointing to the same, and there is nothing in the adapter afaic see that would cause this. Can you have a look?

[rprouse]

@ADBrinkman It doesn't look like the logs you attached show the problem. In that test run packages.config changed so all tests were run.

2019-01-29T15:56:56.2906485Z ***************************** Section 'Get Files that were changed' Starting *****************************
2019-01-29T15:56:56.8634806Z Baseline build: http://sdevtfs:8080/tfs/_permalink/_build/index?collectionId=d2c1e7eb-5f2d-4186-b5f9-6885e90f9e18&projectId=a5fd4a2f-f9c2-4483-9007-f9e2d2002887&buildId=59194
2019-01-29T15:56:56.8645692Z Baseline build id = 59194
2019-01-29T15:56:57.0066518Z Following files were changed in between current build and baseline build.
2019-01-29T15:56:57.0067050Z --------------------------------------------------------
2019-01-29T15:56:57.0067192Z /Projects/Bygga/TotalSpeech 3/TIA/TIANunit/TIATest/packages.config (Could not reason about changes in this file type. Hence all tests will be run.)
2019-01-29T15:56:57.0067238Z /Projects/Bygga/TotalSpeech 3/TIA/TIA/TIA/StringUtil.cs
2019-01-29T15:56:57.0079476Z Changes to unknown file types exist in this build. All tests will be run.
2019-01-29T15:56:57.0081292Z Total files changed = 2
2019-01-29T15:56:57.0686213Z ***************************** Section 'Get Files that were changed' Ended ********************************

I'd like to see the discovery for just changes to one file so that we can see the behavior.

[ADBrinkman]

I believe I was incorrect and adding the CollectDataForEachTestSeparately to the runsettings does resolve the impact test analysis. The problem following this change was actually in my testing - apologies!
My testing all looks to be working as expected now.

Thanks for looking in to this @OsirisTerje, @rprouse

[OsirisTerje]

Very good to hear! Thanks @ADBrinkman !