Issue with configuration file reqnroll.json

[d4v1dk0n7]

Reqnroll Version

2.1.0

Which test runner are you using?

NUnit

Test Runner Version Number

4.2.2

.NET Implementation

.NET 8.0

Test Execution Method

Visual Studio Test Explorer

Content of reqnroll.json configuration file

{
"$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",
"language": {
"feature": "en-US",
"binding": "en-US"
},
"generator": {
"allowDebugGeneratedFiles": false,
"allowRowTests": true,
"addNonParallelizableMarkerForTags": [ "nonparallelizable" ]
},
"runtime": {
"missingOrPendingStepsOutcome": "Pending",
"obsoleteBehavior": "Warn",
"stopAtFirstError": true
},
"trace": {
"stepDefinitionSkeletonStyle": "CucumberExpressionAttribute",
"coloredOutput": true
},
"bindingAssemblies": [
{
"assembly": "QA.TestCore"
}
]
}

Issue Description

Hi team,
First of all I would like to thank you for the incredible work you're doing!! Reqnroll is a fantastic tool :)

I've noticed that the setting "stopAtFirstError" is not working as expected, when I execute the tests and some step fails it continues with next step anyway, even if you have configured the "stopAtFirstError" setting as "true".
As far as I understand with this setting activated once one step fails it shouldn't execute the next... Or I'm missunderstanding how this works?

If I'm wrong please feel free to close the ticket.
Thanks!

Steps to Reproduce

Just executing the tests you'll see that aren't stopping in the first error.

Link to Repro Project

No response

[gasparnagy]

I cannot reproduce the problem. I have used a simple calculator demo app with the following scenario (both Then steps fail):

Scenario: Multily with zero (wrong)
    Given the first number is 5
    And the second number is 0
    When the two numbers are multiplied
    Then the result should be 1
    Then the result should be 123

Without any config, it "continues", so I see the test log as

-> Loading plugin W:\Reqnroll\SampleProjects\ReqnrollCalculator\ReqnrollCalculator.Specs\bin\Debug\net8.0\Reqnroll.MSTest.ReqnrollPlugin.dll
-> Loading plugin W:\Reqnroll\SampleProjects\ReqnrollCalculator\ReqnrollCalculator.Specs\bin\Debug\net8.0\ReqnrollCalculator.Specs.dll
-> Using reqnroll.json
Given the first number is 5
-> done: CalculatorStepDefinitions.GivenTheFirstNumberIs(5) (0.0s)
And the second number is 0
-> done: CalculatorStepDefinitions.GivenTheSecondNumberIs(0) (0.0s)
When the two numbers are multiplied
-> done: CalculatorStepDefinitions.WhenTheTwoNumbersAreMultiplied() (0.0s)
Then the result should be 1
-> error: Expected _calculator.GetResult() to be 1, but found 0 (difference of -1). (0.1s)
Then the result should be 123
-> skipped because of previous errors

But if I add a config file as

{
  "$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",

  "runtime": {
    "stopAtFirstError": true
  }
}

Then the output changes to (note the missing second Then step):

-> Loading plugin W:\Reqnroll\SampleProjects\ReqnrollCalculator\ReqnrollCalculator.Specs\bin\Debug\net8.0\Reqnroll.MSTest.ReqnrollPlugin.dll
-> Loading plugin W:\Reqnroll\SampleProjects\ReqnrollCalculator\ReqnrollCalculator.Specs\bin\Debug\net8.0\ReqnrollCalculator.Specs.dll
-> Using reqnroll.json
Given the first number is 5
-> done: CalculatorStepDefinitions.GivenTheFirstNumberIs(5) (0.0s)
And the second number is 0
-> done: CalculatorStepDefinitions.GivenTheSecondNumberIs(0) (0.0s)
When the two numbers are multiplied
-> done: CalculatorStepDefinitions.WhenTheTwoNumbersAreMultiplied() (0.0s)
Then the result should be 1
-> error: Expected _calculator.GetResult() to be 1, but found 0 (difference of -1). (0.2s)

Could you please double check:

• If the config file is at the right position (root folder of the project)?
• That you have rebuild the project after changing the config file (sometimes VS does not pick up changes if you only change the config file)?
• Maybe try to make a small repro project

[d4v1dk0n7]

Hi @gasparnagy,

I've recreated the specflow project and now all is working fine (I don't know why, because I already tried to rebuild the project previously and the config file was already in root folder).

Anyway now it's working...
Thanks for your help! :)

[gasparnagy]

@d4v1dk0n7 no prob.

Closing the ticket.