Debug.Assert in test method crashes full process #3955
Comments
|
This is by design, Debug.Assert behaves like this, if you are not debugging. I don't think it is great experience (especially for tests) because either:
To prevent this you need to register a trace listener and handle the error there. I did that in VSTest previously, but admittedly my solution is not great because it will translate the assert to exception that can be handled. Instead, it would be better, if the listener threw exception, but also told the test that it must fail, even if the underlying code handles the exception. E.g. using async local storage to pick up the state of the current test. |
|
Just a quick note that I simplified the customer problem for the sake of the repro. In their case, the |
|
We use Debug.Assert in our code base to validate that the desired state was reached. If not, this Debug.Assert is thrown. This is used in both tests and when running code outside of tests. But when our debug code throws this exception during tests, the test host is killed and our tests cannot continue. In NUnit (where we are migrating from), the test would be marked as failed and other tests continue. |
|
I don't see any mention of NUnit handling the Assert calls, do you happen to know where they do it? Running via nunit3-console it crashes the child dotnet.exe process as well.
I think you might be referring to running NUnit tests with VSTest (e.g. with dotnet test, in VS, or with vstest.console). In those cases the solution described in the comment above is used, which will prevent the process from crashing.
But it also does not prevent the assertion exception from being handled, which I think is incorrect. The test below will pass, but it should fail. [Test]
public void Test1()
{
try
{
Debug.Fail("OOOOOOOOOOOOOOOOh!");
}
catch
{
}
} |
|
@Evangelink I think this needs more design to solve properly, would you be willing to mark it as feature and move it to 3.7? To solve this properly we need to establish some context for the test that is accessible globally (or based on async local storage) and put the data about failure there. If this is done, such storage is useful for other things as well, e.g. for soft assertions. |
|
@Liamdoult if this is a blocker for you, you can simply grab the implementation from vstest, and just use Assembly initialize to put it in place. VSTest does just that, but built-in: // file Test1.cs
using Microsoft.VisualStudio.TestPlatform.TestHost;
using System.Diagnostics;
namespace TestProject93
{
[TestClass]
public sealed class Test1
{
[TestMethod]
public void TestMethod1()
{
Debug.Fail("OOOOOh");
}
[AssemblyInitialize]
public static void Setup(TestContext _)
{
TestHostTraceListener.Setup();
}
}
}// file DebugAssertListener.cs
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT license. See LICENSE file in the project root for full license information.
namespace Microsoft.VisualStudio.TestPlatform.TestHost
{
#if NETCOREAPP
using System;
using System.Diagnostics;
using System.Linq;
using System.Reflection;
internal class TestHostTraceListener : DefaultTraceListener
{
public static void Setup()
{
// in the majority of cases there will be only a single DefaultTraceListener in this collection
// and we will replace that with our listener, in case there are listeners of different types we keep
// them as is
for (var i = 0; i < Trace.Listeners.Count; i++)
{
var listener = Trace.Listeners[i];
if (listener is DefaultTraceListener)
{
Trace.Listeners[i] = new TestHostTraceListener();
}
}
}
public override void Fail(string message)
{
throw GetException(message);
}
public override void Fail(string message, string detailMessage)
{
throw GetException((message + Environment.NewLine + detailMessage));
}
public static void ShowDialog(string stackTrace, string message, string detailMessage, string _)
{
var text = !string.IsNullOrEmpty(message)
? !string.IsNullOrEmpty(detailMessage)
? (message + Environment.NewLine + detailMessage)
: message
: null;
throw GetException(text);
}
private static DebugAssertException GetException(string message)
{
var debugTypes = new Type[] { typeof(Debug), typeof(Trace) };
var stack = new StackTrace(true);
var debugMethodFound = false;
var frameCount = 0;
MethodBase method = null;
foreach (var f in stack.GetFrames())
{
var m = f.GetMethod();
var declaringType = m?.DeclaringType;
if (!debugMethodFound && (declaringType == typeof(Debug) || declaringType == typeof(Trace)))
{
method = m;
debugMethodFound = true;
}
if (debugMethodFound)
{
frameCount++;
}
}
var stackTrace = string.Join(Environment.NewLine, stack.ToString().Split(Environment.NewLine).TakeLast(frameCount));
var methodName = method != null ? $"{method.DeclaringType.Name}.{method.Name}" : "<method>";
var wholeMessage = $"Method {methodName} failed with '{message}', and was translated to {typeof(DebugAssertException).FullName} to avoid terminating the process hosting the test.";
return new DebugAssertException(wholeMessage, stackTrace);
}
}
internal sealed class DebugAssertException : Exception
{
public DebugAssertException(string message, string stackTrace) : base(message)
{
StackTrace = stackTrace;
}
public override string StackTrace { get; }
}
#endif
} |
Evangelink
modified the milestones:
MSTest 3.6.2 / Platform 1.4.2,
MSTest 3.7 / Platform 1.5
Evangelink
modified the milestones:
MSTest 3.7 / Platform 1.5,
MSTest 3.8 / Platform 1.6
Describe the bug
Debug.Assert in test method crashes full process
Steps To Reproduce
Output:
Expected behavior
No issue
Actual behavior
Process crashes
The text was updated successfully, but these errors were encountered: