Running tests using the VS Solution results in an exception if any projects target .NET Standard

[rameshkadev]

Hi,

When running nunit tests through nunit.console version 3.15 generates NUnit.Engine.NUnitEngineException : Unsupported Target Framework: .NETStandard,Version=v2.0

I've a solution file with two class library projects,

1. Main library targets framework version .Net Standard 2.0
2. Test library (contains Nunit tests v3.13.3.0) targets framework version .Net framework 4.8

When I run the below command:
C:\Program Files (x86)\NUnit.org\nunit-console>nunit3-console.exe "C:\source\repos\Project1\Project1.sln" --config=Debug
Here is the output with NUnit.Engine.NUnitEngineException : Unsupported Target Framework: .NETStandard,Version=v2.0

NUnit Console 3.15.0 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
10 May 2022 12:57:15

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    C:\source\repos\Project1\Project1.sln

NUnit.Engine.NUnitEngineException : Unsupported Target Framework: .NETStandard,Version=v2.0

--NUnitEngineException
Unsupported Target Framework: .NETStandard,Version=v2.0
   at NUnit.Engine.Services.RuntimeFrameworkService.SelectRuntimeFrameworkInner(TestPackage package)
   at NUnit.Engine.Services.RuntimeFrameworkService.SelectRuntimeFrameworkInner(TestPackage package)
   at NUnit.Engine.Services.RuntimeFrameworkService.SelectRuntimeFrameworkInner(TestPackage package)
   at NUnit.Engine.Services.RuntimeFrameworkService.SelectRuntimeFramework(TestPackage package)
   at NUnit.Engine.Runners.MasterTestRunner.GetEngineRunner()
   at NUnit.Engine.Runners.MasterTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)

Attached sample project for your reference:
Project1.zip

[CharliePoole]

This is likely to be a problem with the Visual Studio project loader extension. What version of it are you using with NUnit-console?

In addition, you should try removing the reference to the nunit3 test adapter, at least temporarily. The adapter brings in it's own version of the nunit engine, which may be different from the 3.15 version used by the console runner.

[rameshkadev]

Nunit-console is using vs-project-loader.dll v3.9.0.0

Also, I've temporarily removed nunit3 test adapter but still the same issue.

[CharliePoole]

@rameshkadev I'm confirming that this is a bug in both the engine and the project loader.

Running your .NET Standard main assembly directly or via the .csproj file gives the same result. In that case the error is correct as it tells you that netstandard tests are not supported - although the message could be a bit clearer!

When running via the solution, your intention is obviously to only run test assemblies. However, in order to determine whether something is a test assembly, the engine must first load the assembly and examine it for tests.

It could be resolved in the VS project loader extension by not including any .NET Standard assemblies in the expantion of the solution, which is provided to the engine. In fact, I'll create an issue for the loader to do exactly that.

This can also be resolved in the engine by remembering that the assembly originated from expansion of a VS solution. In that case, we can merely ignore the assembly rather than giving an error. This is a slightly less desirable solution but has the benefit that it will work without upgrading the version of the project loader, which the user has installed.

UPDATE: I used a different approach to resolve this. See following comment.

As a workaround for the time being, you should just run your test assembly or .csproj directly. If you have multiple test assemblies in the solution, you can create an nunit project containing only those assemblies.

[CharliePoole]

Thinking about this, I wondered why we can't allow running tests, which target .NET Standard, which has been a stated limitation ever since we began to support .NET Core. The NUnit framework actually provides a .NET Standard 2.0 build, which is used for all .NET Core test assemblies.

I modified the code so that any .NET Standard test is automatically loaded using the .NET Core 3.1 engine. With a few added corrections, this builds, tests and creates packages without any problem. It also resolves the current issue and provides a cleaner error message (no tests found) for the .NET Standard non-test assembly provided by @rameshkadev.

I'm creating a new issue to further explore what it will take to fully support tests targeting various versions of .NET Standard.

[CharliePoole]

This issue has been resolved in version 3.16.0

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org

[rameshk553]

@CharliePoole, thanks for the fix, it is working as intended in version 3.16.0

[janis-veinbergs]

Why is 3.16.0 not available in NuGet? Last release I see there is 3.13.3: https://www.nuget.org/packages/NUnit/

[CharliePoole]

@janis-veinbergs That link is the NUnit framework, not the NUnit console runner, which is here: https://www.nuget.org/packages/NUnit.ConsoleRunner