NUnit failed to load .exe with uninformative error message in test discovery

[apriori]

Extra information:

Visual Studio : 2012 Update 5
The project is using .Net 4.5, NUnit 3.4.1. It is a mixed mode application, using C#, MC++ and C++/CLI. The project in question that is not loaded is 100% MC++.

Do you have any pointers on where to search for the error? Also, let me also suggest to improve the error messages. The mere information that something fails is useless without some extra information.

[rprouse]

How are you running your tests, VS or the console runner. Are all the projects x86 or x64? What is the unhelpful error, it might mean something to us.

Your scenario works, some of our unit tests at work are the same, but the native code forces you to be more correct.

[apriori]

Hello, thank you for your reply. I try to run it via the VS test runner using the Addin, but detection fails. All projects are x64, also the C# projects are specifically set to x64.

Unfortunately "NUnit failed to load $PathToExe" is exactly the only error I get in the output window from the Unit Test Adapter. That is why I called it unhelpful :)

[CharliePoole]

The specific sequence of messages in the Output Window would tell us whether this occurs in discovery or execution, but given the content it actually doesn't matter.

The message is actually very informative. It means that NUnit was instructed to load an assembly named "$PathToExe", and could not do so, which is rather unsurprising. :-)

I gather that $PathToExe is the name of some property that you are expecting to be populated with a value, but which is not.

[apriori]

Ohm, no, its not a property or anything. $PathToExe was simply used by me as a placeholder for a valid path to the compiled .exe of my application.

------ Discover test started ------
NUnit Adapter 3.4.1.0: Test discovery starting
NUnit failed to load C:\project\x64\Debug\application.exe
Assembly contains no NUnit 3.0 tests: D:\project\somedll.dll

NUnit Adapter 3.4.1.0: Test discovery complete
========== Discover test finished: 0 found (0:00:22,4702468) ==========

[rprouse]

Did you set the Visual Studio test execution to 64 bit? Sorry, greyed out because I don't have a project loaded, but this is where it is.

[rprouse]

I have categorized this as a question for now. We can change it if we find issues, but I think it is likely a setup problem.

[apriori]

@rprouse Yes I did. I was falling over that trap with a Boot Test Adapter already :)

[rprouse]

Are you using the NuGet package of NUnit or the MSI install? There is a bug in the MSI install where the 4.5 reference goes to the portable version of the framework which doesn't run under VS.

[CharliePoole]

@apriori The message you have is a sort of "last resort" message, after a number of things have been checked. It's not worded well, I'm afraid.

In fact, the assembly was actually loaded. What failed was loading tests. The assembly loaded and discovery finished without any exceptions being thrown, but the assembly is non-runnable. The code next checks to see if that's because no tests were found. It would display a different message in that case, as it does in the next line of output for a different dll. Finally, it gives up and displays the message you see. It should probably instead check the XML for an embedded message left behind by the framework.

If you try running this same assembly as a test under nunit3-console, you may get a better message that will explain things. OTOH, you may not, since nunit3-console will try to run the assembly in an appropriate process, whereas the adapter uses the process given to it by Visual Studio.

[CharliePoole]

@rprouse I think this is at least a message bug, so I'm going to re-label it. Looking at the code, we actually examine the message in the XML to see if the assembly is non-runnable because it doesn't have tests. However, we never use the message we found and just give this generic message.

[CharliePoole]

I'm leaving this as both a bug and a question, since we should still try to resolve @apriori 's problem.

[apriori]

I use the nuget package. Although I had to manually reference the .net 4.5 nunit.framework.dll for the project.

Well, using nunit3-console I get a tad bit more useful message:

1) Invalid : C:\project\x64\Debug\application.exe
Die Datei oder Assembly "application, Version=1.0.6115.30727, Culture=neutral, PublicKeyToken=null" oder eine Abhängigkeit davon wurde nicht gefunden. Es wird versucht, eine nicht überprüfbare ausführbare Datei mit Fixups zu laden (IAT mit mehr als 2 Abschnitten oder ein TLS-Abschnitt). (Ausnahme von HRESULT: 0x80131019)

Sry, I kind of failed trying to set the codepage of cmd to english. This is a typical load error, although "Fixups" abd (IAT/TLS) might give a hint.

Full TestResult.xml

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-run id="2" testcasecount="0" result="Failed" label="Invalid" total="0" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="3.4.1.0" clr-version="4.0.30319.42000" start-time="2016-09-28 16:13:14Z" end-time="2016-09-28 16:13:14Z" duration="0.084194">
  <command-line><![CDATA["C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe"  "C:\project\x64\Debug\application.exe" --work=D:]]></command-line>
  <test-suite type="Assembly" id="0-1000" name="application.exe" fullname="C:\project\x64\Debug\application.exe" runstate="NotRunnable" testcasecount="0" result="Failed" label="Invalid" site="SetUp" start-time="2016-09-28 16:13:14Z" end-time="2016-09-28 16:13:14Z" duration="0.023630" total="0" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0">
    <environment framework-version="3.4.1.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.1.7601 Service Pack 1" platform="Win32NT" cwd="C:\" machine-name="SOME-MACHINE" user="Some.User" user-domain="SOME-DOMAIN" culture="de-DE" uiculture="de-DE" os-architecture="x64" />
    <settings>
      <setting name="WorkDirectory" value="D:" />
      <setting name="ImageRuntimeVersion" value="4.0.30319" />
      <setting name="ImageRequiresX86" value="False" />
      <setting name="ImageRequiresDefaultAppDomainAssemblyResolver" value="False" />
      <setting name="NumberOfTestWorkers" value="8" />
    </settings>
    <properties>
      <property name="_SKIPREASON" value="Die Datei oder Assembly &quot;application, Version=1.0.6115.30727, Culture=neutral, PublicKeyToken=null&quot; oder eine Abhängigkeit davon wurde nicht gefunden. Es wird versucht, eine nicht überprüfbare ausführbare Datei mit Fixups zu laden (IAT mit mehr als 2 Abschnitten oder ein TLS-Abschnitt). (Ausnahme von HRESULT: 0x80131019)" />
    </properties>
    <failure>
      <message><![CDATA[Die Datei oder Assembly "application, Version=1.0.6115.30727, Culture=neutral, PublicKeyToken=null" oder eine Abhängigkeit davon wurde nicht gefunden. Es wird versucht, eine nicht überprüfbare ausführbare Datei mit Fixups zu laden (IAT mit mehr als 2 Abschnitten oder ein TLS-Abschnitt). (Ausnahme von HRESULT: 0x80131019)]]></message>
    </failure>
  </test-suite>
</test-run>

Thank you very much for your help guys. I continue on this tomorrow.

[MaxKot]

I have a similar issue.

I have a solution with both managed and native projects. NUnit adapter seems to try to load a native DLL and fails whenever I rebuild the project. You can see the solution here: Log4JDash.

Error message (Log4JParserC.dll is an unmanaged library):

------ Discover test started ------
NUnit Adapter 3.5.1.0: Test discovery starting
Error: Unable to get runner for this assembly. Check installation, including any extensions.
BadImageFormatException: Format of the executable (.exe) or library (.dll) is invalid.
Assembly not supported: D:\develop\Log4JDash\src\Debug\Log4JParserC.dll
NUnit Adapter 3.5.1.0: Test discovery complete
========== Discover test finished: 0 found (0:00:00,0570033) ==========

Visual Studio info:

Microsoft Visual Studio Community 2015
Version 14.0.25431.01 Update 3
Microsoft .NET Framework
Version 4.6.01055

[anttirt]

I have the same issue as MaxKot; it's related to #248.

The issue is otherwise benign except that the Error log causes focus to switch to the Tests section in the output window after every build of the solution, which can be very annoying. :)

The NUnit runner should somehow detect that a project is a native code project and skip it without emitting an error message, but I don't know if there's an API for that off the top of my head. The built in test discoverer for Visual Studio has the following to say:

Test run will use DLL(s) built for framework Framework45 and platform X86. Following DLL(s) will not be part of run: 
my_native_project.exe is built for Framework None and Platform X64.

So perhaps skipping "assemblies" with Framework = None would do the trick?

I don't have time to experiment with this right now but may take a look at it later this week.

[CharliePoole]

@anttirt @MaxKot You both seem to have a different problem from that which was originally reported.

When I marked this easyfix, hoping somebody would take it on, we had ascertained that the original problem could be resolved by simply giving a better message - the message that was already reported internally in the engine.

However, that will not give you guys anything but the same messages you already have. The engine currently has no awareness of C++ assemblies. @ANTIRT 's suggestion seems like a good one. Could you file another issue asking us to check for Framework None? Finding the right place to do that in the engine most likely takes this out of the "easyfix" category for newbies.

[MaxKot]

@CharliePoole I filed issue #302. Sorry it took me so long.

[OsirisTerje]

In later versions we display this as Warnings, not errors.
That should be good enough.
We could move them under a verbosity setting, if there are more requests for this.