Error running tests from assembly built using VS2017 csproj file format

[ManfredLange]

Steps to reproduce:

1. In Visual Studio 2017 create a new project to create a new project of type "Class library (.NET Standard)" to contain your tests
2. Edit project file and change the value of from "netstandard2.0" to "net461"
3. Add nuget package "NUnit"
4. Add another nuget package, e.g. Autofac, and add code that reference a symbol in that other package
5. Add one or more tests to the test project
6. Build and execute tests using NUnit Console

This fails with FileLoadException because the files of the other nuget package, e.g. Autofac, cannot be loaded. They are not copied to TargetDirectory during build. [Update 16 Sep 2017: I stand corrected: the files are copied but nunit-console still falls over with the FileLoadException. I'll re-run the scenario to confirm again and rule out other causes.]

Step 1 can be replaced with using "dotnet new classlib" at a command prompt or in a powershell.

Environment

Visual Studio 2017 version 15.4 with packages NUnit 3.8.1, NUnit.ConsoleRunner 3.7.0 and Microsoft.Net.Compilers 2.3.2 and using dotnet version 2.0.0.

Expected Behavior

NUnit engine should resolve and load assemblies that the test assembly may depend on, e.g. using the information from obj/project.assets.json of the test project created in step 1 above.

Notes

This may already be filed as an issue. I searched existing issues but couldn't find anything obvious. I apologize if this is a duplicate.

[rprouse]

Excellent report, thanks. As a temporary workaround, although not a great one, you should be able to use the Visual Studio adapter and the dotnet test command.

[ManfredLange]

@rprouse Thank you for the suggestion, I'll give that a try.

[CharliePoole]

@ManfredLange @rprouse Maybe I'm behind the curve. Can one of you explain why this is a bug? NUnit is supposed to work from assemblies and only assemblies. It does nothing to load any assembly but the first one and all resolution is done by the runtime or - in the case of dynamically loaded assemblies - by the appication itself.

If this were to be implemented, how would you run tests deployed for QC for example?

[ManfredLange]

Yes, I agree, it should be able to work off an assembly only. However, with the change in the tooling it seems that it is not sufficient to simply point nunit3-console to the assembly. Execution will fail with the FileLoadException reported in this issue.

I just reproduced this issue once more, this time building the project using "dotnet build". It builds fine, just like building from VS2017. Those two build options appear to do the same.

I also tried "dotnet publish" (with no further options), which just copies everything into a target folder but unfortunately does't solve the issue.

Both scenarios, with and without publish, have the dependent packages files right in the folder of the test assembly (eg bin/Debug/net461). Still the runtime doesn't load them automatically for some reason. Perhaps it treats them as "private paths"?

I then tried the following: I wrote a little console app targeting net461 (yup that works) and added some code that loads the test assembly. It retrieves the tests from the assembly (using reflection based on full type names) and then executes them. No need for a NUnit reference in the console app. However, I had to add an AssemblyResolve event handler that loaded the assembly (ie the dependency) sitting right next to the test assembly. Without the event handler my little prototype ran into the same problem. With the event handler present it worked just fine.

In my opinion this small experiment seems to suggest that either the console or the engine may need an AssemblyResolve event handler that can handle this scenario. Or - another option - it may require a different type of AppDomain setup although we need to be aware that the AppDomain concept is not the same under .NET Core.

For now I'll be using "dotnet test" which seems to do the work. Unfortunately this comand doesn't like the solution file because that runs into a different bug unrelated to NUnit (and already reported and worked on someplace over at the msbuild guys). I already have a workaround in place for that other, unrelated bug.

[rprouse]

@CharliePoole, the issue is that Visual Studio doesn't copy dependent assemblies into the bin directory when it builds using the new project format.

[CharliePoole]

Understood. That was my points to why it shouldn't be seen as an NUnit issue.

[ManfredLange]

I'll revisit this issue as it appears as if there is yet another factor at play. I need to investigate this further.

Update: Nope, false positive. This issue remains to be reproducible using the steps to reproduce despite the file in question actually actually being present in the output directory.

Here is are the assemblies in play in this scenario:

1. Test assembly ("test assembly") containing the tests and a reference to nuget package NUnit. Uses new csproj file format targeting net461 and PackageReference elements within the project file.
2. A second assembly ("assembly under test") referenced by the test assembly. "assembly under test" has a dependency on NuGet package "Autofac" version 4.6.1. "assembly under test" is using old csproj file format and also packages.config. "assembly under test" uses value "v4.6.1" for the targetframeworkversion element in the project file
3. Executing nunit-console with the name and path of the project file of the "test assembly" fails with FileLoadException for "Autofac"
4. Executing nunit-console with the name and path of output file of the "test assembly" fails as well with FileLoadException for "Autofac" even though the assembly Autofac.dll is present.

So it's testAssembly -> assemblyUnderTest -> Autofac (all targeting .NET Framework 4.6.1)

The full text of the exception for steps 3 and 4 is identical and is as follows:

System.IO.FileLoadException : Could not load file or assembly 'Autofac, Version=4.0.0.0, Culture=neutral, PublicKeyToken=17863af14b0044da' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

Next, I added the packages "NUnitTest3Adapter" and "Microsoft.NET.Test.Sdk". As suggested "dotnet test" now works which is great.

But here is the surprise: nunit-console works now as well even though none of the two of them references "Autofac".

Next, I removed package "Microsoft.NET.Test.Sdk" and nunit-console fails again with FileNotFoundException

To make a cross-check I added "Microsoft.NET.Test.Sdk" back in and removed "NUnit3TestAdapter". nunit-console works again.

So the work around for the reported issue this: add package "Microsoft.NET.Test.Sdk" and nunit-console will find assemblies that are already in the output dir - "Autofac" in my scenario.

[ManfredLange]

Assuming in my quick check I found the correct line of code there appears to be at least one instance where package Microsoft.NET.Test.Sdk adds an AssemblyResolve event handler. This could potentially explain why nunit-console works with this package referenced and fails while the package is absent.

It may also point towards a possible solution in either nunit-console or the nunit engine. Perhaps an appropriate AssemblyResolve event handler would fix the issue. I used this approach as well in my little console app prototype and that executed the tests without the FileLoadException.

[rprouse]

That was my points to why it shouldn't be seen as an NUnit issue.

I would agree in principle, but we aren't going to change the way that .NET builds. We also support it with the dotnet test command because of the Microsoft.NET.Test.Sdk resolve handler (thanks for finding that @ManfredLange, I was actually wondered how it worked for .NET Core projects 😄 )

I wonder however if adding the Microsoft.NET.Test.Sdk is the correct solution to this, not the workaround. We know it is required for .NET Core/Standard tests, but it might be more correct to say that it is required for tests that use the new CSPROJ format.

This could be a documentation issue.

[ManfredLange]

My two cents: I think adding Microsoft.NET.Test.Sdk is a workaround for the new csproj format. Based on the experiments it appears that all that is needed is an appropriate AssemblyResolve event handler.

Everything else coming with Microsoft.NET.Test.Sdk may not be needed. But then again, that package may not be too much of a burden in the first place. If that is the case, perhaps making it a dependency for NUnit for projects in the new csproj format might be the best option.

[CharliePoole]

@ManfredLange I would have no problem with using an AssemblyResolve handler if we could do it well. What I'm not clear on is how NUnit can know where to look for the assembly in order to resolve it. The engine has no knowledge that it is running in an output directory created by VS and certainly doesn't know what project format was used. Some application that's calling the engine has to tell it. That makes sense to me if the runner is the VS adapter but not so much sense in the general case. However, I may be missing something so feel free to fill me in. 😄

BTW, we already use an AssemblyResolve event handler for certain things, so if this is done it should probably be added to that one.

[CharliePoole]

@ManfredLange BTW - in case you were not aware - I'm on vacation and many of my limited responses are via my cell. Hence, the infrequency and lack of detail in my posts. 😄

[CharliePoole]

Needs to be re-confirmed using the 4.0 codebase.

[CharliePoole]

@ManfredLange I'm attempting to clean up very old issues like this one. Although it hasn't been worked on explicitly, SDK-format projects are now supported and various other issues are resolved in the latest 3.16 development builds on MyGet.

I think this is probably still valid, but it will help if you can try your case using one of the dev builds. You should use the latest 3.16 build rather than any of the 4.0 builds - I'll be porting changes to main (and 4.0.--devxxxxx) after the 3.16 release.

[CharliePoole]

This issue has been resolved in version 3.16.0

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org