.NETStandard test assemblies are not supported by this version of the engine

[rcocks-hl]

Hi,

I'm having an issue where trying to running nunit3-console is failing with the error .NETStandard test assemblies are not supported by this version of the engine.

command "".\..\NuGet.packages\NUnit.ConsoleRunner.3.10.0\tools\nunit3-console.exe" --skipnontestassemblies --noresult  --config=Debug --where "cat == Unit" "<solution>.sln""

However the tests themselves are all framework, and the .net standard assembly it complains about isn't a test assembly, but merely referenced from the (.net Framework) test assembly.

It is my understanding that .net framework should be able to reference .net standard, and I expected nunit3-console to support tests which are written in framework test project referencing a .net standard assembly.

The unit tests run fine from visual studio and it is only the console runner that is having an issue.

[CharliePoole]

Does your .NET Standard assembly reference the NUnit framework? NUnit's definition of a "test assembly" is any assembly, which references the framework. If such an assembly is under your control, you can make it into a non-test assembly by use of the NonTestAssembly attribute.

[rcocks-hl]

Hi Charlie,

Our .Net Standard assemblies do not reference nunit, which is why I'm finding it odd that this error is occurring.

[CharliePoole]

I notice you are using 3.10, which is quite old. Can you use a newer version? Preferably 3.15.0?

[rcocks-hl]

I noticed the same and so I've just updated to 3.15.2 and get a similar, though slightly different error:

 NUnitSolutionUnit:
     ".\..\NuGet.packages\NUnit.ConsoleRunner.3.15.2\tools\nunit3-console.exe" --skipnontestassemblies --noresult
     --config=Debug --where "cat == Unit and cat != Performance" "<solution>.sln"
     NUnit Console 3.15.2 (Release)
     Copyright (c) 2022 Charlie Poole, Rob Prouse
     30 June 2022 16:21:47

     Runtime Environment
        OS Version: Microsoft Windows NT 6.2.9200.0
        Runtime: .NET Framework CLR v4.0.30319.42000

     Test Files
         <solution>.sln

     Test Filters
         Where: cat == Unit and cat != Performance

     NUnit.Engine.NUnitEngineException : Unsupported Target Framework: .NETStandard,Version=v2.0

     --NUnitEngineException
     Unsupported Target Framework: .NETStandard,Version=v2.0
        at NUnit.Engine.Services.RuntimeFrameworkService.SelectRuntimeFrameworkInner(TestPackage package)
        at NUnit.Engine.Services.RuntimeFrameworkService.SelectRuntimeFrameworkInner(TestPackage package)
        at NUnit.Engine.Services.RuntimeFrameworkService.SelectRuntimeFrameworkInner(TestPackage package)
        at NUnit.Engine.Services.RuntimeFrameworkService.SelectRuntimeFramework(TestPackage package)
        at NUnit.Engine.Runners.MasterTestRunner.GetEngineRunner()
        at NUnit.Engine.Runners.MasterTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
        at NUnit.Engine.Runners.MasterTestRunner.Run(ITestEventListener listener, TestFilter filter)
        at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)

Unfortunately this new error no longer says which assemblies it finds unsupported.

[CharliePoole]

OK. I'll look at the code. It may be that the check is in the wrong place, since every assembly has to be examined in order to determine if it's a test assembly. Meanwhile, I suggest defining and adding your own attribute with the full name NUnit.Framework.NonTestAssemblyAttribute and applying it. The runner checks for the attribute by name. :-) Of course, this won't work if the check is made before the attribute is looked for. :-(

[rcocks-hl]

Just to note I edited my previous comment to clarify about the error, sorry if that caused confusion.

[rcocks-hl]

Hi @CharliePoole ,

After upgrading to 3.15 from 3.10 to try to debug this issue, our unit and integration tests in our main branch (which does not have this .net standard project) went from taking under 20 minutes to timing out after 40 minutes.

Do you want me to raise that performance issue as a separate issue, or are there some settings differences (such as parallelism?) that might have caused that?

Regarding this issue, if time permits I'll try to build and run nunit-console to catch where exactly we're seeing the error and whether we can recreate a minimum test case for you without our full solution.

Kind regards,

[rcocks-hl]

Hi @CharliePoole ,

Just to let you know the performance regression was due to a parallism change which we have resolved.

On the initial error, I am working on reproducing a small test case although I am finding it difficult to run my own custom built version because I am having trouble getting it to load the vsprojectloader addon which it requires.

[CharliePoole]

@rcocks-hl If you are building from current source, rather than the 3.15 source, then the project loader extension won't work. It has not yet been updated to use the 4.0 interface. If you're using the 3.15 source, it ought to work.

[rcocks-hl]

Hi @CharliePoole ,

Thank you for pointing out that the version 4 does not support the visual studio loader.

Using the 3.15.2 release branch I still had trouble loading the addins when trying to run the build that outputs to the net6.0 folder but was able to reproduce the problem using the build that outputs to the net35 folder.

In that, I get the same error as when I run with the 3.15.2 release nuget package. Debugging it, it looks like this block in RuntimeFrameworkService is the culprit:

switch (frameworkName.Identifier)
{
    case ".NETFramework":
        targetRuntime = RuntimeType.Net;
        break;
    case ".NETCoreApp":
        targetRuntime = RuntimeType.NetCore;
        break;
    default:
        throw new NUnitEngineException("Unsupported Target Framework: " + imageTargetFrameworkNameSetting);
}

The frameworkName.identifier reports .NETStandard,Version=v2.0.

We find it surprising that .NETStandard is not supported by this given that .net standard is designed to be loaded by both framework and core.

Also to be clear in this case, it is not the test assemblies themselves that are .NETStandard, just a dependency of the test assemblies. The tests themselves are in a .net framework assembly.

[CharliePoole]

@rcocks-hl It sounds as if this is still an issue in 3.15.2 then. I'll try to confirm it with an example.

Meanwhile, if you want to run using the solution file, I suggest using the workaround of defining your own NonTestAssembly attribute in the assembly, which is causing a problem.

[rcocks-hl]

HI @CharliePoole,

Did you manage to confirm this bug?

[CharliePoole]

@rcocks-hl Seems that I lost track of this issue but fixed another one, which describes the same problem, #1182. It's in the 3.16 release, which just came out a few hours ago. Give that a try and let me know if it works and if so I'll close this one as well.

[rcocks-hl]

Hi @CharliePoole ,

Thanks for getting back to me. I don't have my reproduction case ready to test right now but reading up on that issue I'm happy for you to close this as a duplicate and I'll create a new issue if I do encounter one in my testing of 3.16.