Visual Studio Test Task ignores Test Filter Criteria with Nunit

[RCerff]

Hello,

I have the problem that Team Services Test task ignores Test Filter criteria.
I'am using Nunit 3.6.1 and Nunit3TestAdapter 3.7.0
I decorate my TestFixture with a category

[TestFixture(Category = "DataAccess")]
public class RetailerDbAccessTestsAsync {...}

As Test Filter criteria I use TestCategory!=DataAccess

It seems that the task ignores this criteria.

2017-05-05T12:29:57.7446530Z ##[section]Starting: Test Assemblies \release*test.dll;-:*\obj**
2017-05-05T12:29:57.7446530Z ==============================================================================
2017-05-05T12:29:57.7446530Z Task : Visual Studio Test
2017-05-05T12:29:57.7446530Z Description : Run tests with Visual Studio test runner
2017-05-05T12:29:57.7446530Z Version : 1.0.84
2017-05-05T12:29:57.7446530Z Author : Microsoft Corporation
2017-05-05T12:29:57.7446530Z Help : More Information
2017-05-05T12:29:57.7446530Z ==============================================================================
2017-05-05T12:29:57.7776654Z Preparing task execution handler.
2017-05-05T12:30:00.8419190Z Executing the powershell script: d:\a_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\1.0.84\VSTest.ps1
2017-05-05T12:30:01.5073344Z Working folder: d:\a\1\s
2017-05-05T12:30:01.5073344Z Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "d:\a\1\s\Trovare.Base\Trovare.Base.Sample.Test\bin\Release\Trovare.Base.Sample.Test.dll" /TestCaseFilter:"TestCategory=DataAccess" /logger:trx /TestAdapterPath:"d:\a\1\s"
2017-05-05T12:30:01.8207382Z Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
2017-05-05T12:30:01.8207382Z Copyright (c) Microsoft Corporation. All rights reserved.
2017-05-05T12:30:01.8207382Z
2017-05-05T12:30:01.9807385Z Starting test execution, please wait...
2017-05-05T12:30:02.2447382Z Information: NUnit Adapter 3.7.0.0: Test execution started
2017-05-05T12:30:02.2447382Z
2017-05-05T12:30:02.2447382Z Information: Running all tests in d:\a\1\s\Trovare.Base\Trovare.Base.Sample.Test\bin\Release\Trovare.Base.Sample.Test.dll
2017-05-05T12:30:02.2447382Z
2017-05-05T12:30:03.5122750Z Information: NUnit3TestExecutor converted 12 of 12 NUnit test cases
2017-05-05T12:30:03.5132756Z

Then all the Test are run.

I've tried Categoy instead of TestCCategory but the it complaines that Category is not a valid value for filter criteria.

I'am missing something? Or what might be the reason. Thank you in advance for any help.

Best Regards
Rolf

[nigurr]

Hi,

Can you run this command in your local machine and see if it's working
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "d:\a\1\s\Trovare.Base\Trovare.Base.Sample.Test\bin\Release\Trovare.Base.Sample.Test.dll" /TestCaseFilter:"TestCategory=DataAccess" /logger:trx /TestAdapterPath:"d:\a\1\s"

If not can you please check by changing this to /TestCaseFilter:TestCategory=DataAccess works?
http://stackoverflow.com/questions/31595179/how-can-i-filter-nunit-tests-when-running-vstest-console-exe

[RCerff]

Hello Nigurr,

thank you for your response. At the moment I'am in holiday. I will try that, when I'am back next week.

Best regards
Rolf

[RCerff]

Hello Nigurr,

I've tried the command above on my local machine with the same result. What I can see is that the filter works partly. It filters out only the tests in one file with the category of DataAccess set. It seems that filter is only applied for one file and the same filter on another file is ignored?

I can see this behavior on my local run and Team Services build.

Regards
Rolf

[nigurr]

@RCerff, sorry didn't get you. There's only one "test dll" available from your build log. It seems that filter is only applied for one file and the same filter on another file is ignored seems to be bit confusing?
Can you let me know what are the test dlls are present, exact command that you have given to vstest.console.exe and expected output? This will simplify confusions :)

[RCerff]

Ah ok. I will try to clarify it.

• I have only one Test assembly (Trovare.Base.Sample.Test.dll)
• In this assembly I have my two test classes, each decorated with [TestFixture(Category = "DataAccess")]. There are also some other classes without or with different Category values.
• Each test class is in its own code file (*.cs file)

I've used the command exactly (with correct paths) as you recommended.

Best Regards
Rolf

[nigurr]

@RCerff thanks for the info. Will add vstest team look into this. @smadala

[smadala]

@RCerff I have created a sample project with two class, one with [TestFixture(Category = "DataAccess")] and other without it. I'm able to use filter. Following is output.

C:\Users\samadala\Documents\Visual Studio 2015\Projects\NUnitTestCaseFilter\NUnitTestProject1\bin\Debug>vstest.console.exe NUnitTestProject1.dll /testadapterpath:C:\Users\samadala\.nuget\packages\nunit3testadapter\3.7.0\tools\ /testcasefilter:"TestCategory=DataAccess"
Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Information: NUnit Adapter 3.7.0.0: Test execution started

Information: Running all tests in C:\Users\samadala\Documents\Visual Studio 2015\Projects\NUnitTestCaseFilter\NUnitTestProject1\bin\Debug\NUnitTestProject1.dll

Information: NUnit3TestExecutor converted 2 of 2 NUnit test cases

Information: NUnit Adapter 3.7.0.0: Test execution complete

Passed   TestMethodWithCategory

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 0.4629 Seconds

C:\Users\samadala\Documents\Visual Studio 2015\Projects\NUnitTestCaseFilter\NUnitTestProject1\bin\Debug>
C:\Users\samadala\Documents\Visual Studio 2015\Projects\NUnitTestCaseFilter\NUnitTestProject1\bin\Debug>vstest.console.exe NUnitTestProject1.dll /testadapterpath:C:\Users\samadala\.nuget\packages\nunit3testadapter\3.7.0\tools\ /testcasefilter:"TestCategory!=DataAccess"
Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Information: NUnit Adapter 3.7.0.0: Test execution started

Information: Running all tests in C:\Users\samadala\Documents\Visual Studio 2015\Projects\NUnitTestCaseFilter\NUnitTestProject1\bin\Debug\NUnitTestProject1.dll

Information: NUnit3TestExecutor converted 2 of 2 NUnit test cases

Information: NUnit Adapter 3.7.0.0: Test execution complete

Passed   TestMethodWithOutCategory

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 0.4280 Seconds

Here is sample Solution I used: NUnitTestCaseFilter.zip.
Please try this on your machine, If it fails, provide us vstest,console.exe TPTrace logs. If it pass Share a sample project which can repro the issue?
On vstest task set System.Debug to true and share logs.
Please also provide TPTrace logs for Trovare.Base.Sample.Test.dll case.

[RCerff]

@smadala

My guess is that the filter works partly in some scenarios.

I think to reproduce my issue change the following:

Add another class with the same Category value [TestFixture(Category = "DataAccess")].

Then run the test again.

Is the filter applied on both classes?

Thank you in advance

Best Regards
Rolf

[smadala]

@RCerff Yes, Filter is applied even after adding another class with Category annotation.
Isn't the same case worked on your local machine?
Are you able to to filter on shared(NUnitTestCaseFilter) project?
Please share TPtrace logs requested above?
Please Share a sample project which repro issue on your local machine?

[RCerff]

At the moment I have a new general problem with vstest.console.exe. With your project and my project I get the following message:
"Warning: No test is available in C:\projects\RTD\Source\Repos\Architecture Blueprint\Trovare.Base\Trovare.Base.Sample.Test\bin\Debug\Trovare.Base.Sample.Test.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again."

And no Test has run.

In Visual Studio it works fine.

Maybe this problem has to do with a Windows 10 Updates installed yesterday - (KB3150513) und (KB4019472)?

Attached the TpTrace.
TpTrace.txt

Regards
Rolf

[AbhitejJohn]

@RCerff : It looks like testadapterpath isn't being passed through. Were you using a similar command-line to what @smadala shared above?

[RCerff]

Yes you were right. I forgot testadapterpath switch. With your Test assembly the filter works as expected, but with my project the filter does not seem work correctly. Also it seems to ignore the Ignore attribute on one of my test methods.

Here a new TpTrace

TpTrace.txt

[AbhitejJohn]

Thanks @RCerff . The logs do not seem to indicate anything odd. However, from what you described this seems related specifically to the nunit adapter. Here would be a good place to get answers to what's wrong here.

[nigurr]

Please open this issue in github.com/Microsoft/vstest. Closing this thread as it's related to vstest.console runner.

[RCerff]

Good news about this issue. This morning I've installed the 3.8.0-alpha1 of the NUnit3TestAdapter. This fixed the issue.

Thank you for your efforts.

Best Regards