Test cases of Scenario Outline not always matched with results when using VsTest

[rik-smeets]

Consider the following Gherkin scenario:

Scenario Outline: Add two numbers
	Given I have entered <number1> into the calculator
	And I have entered <number2> into the calculator
	When I press add
	Then the result should be <number3> on the screen

	Examples: 
	| number1 | number2 | number3 |
	| 1       | 2       | 3       |
	| 1       | 3       | 4       | 

SpecFlow will generate the unit test names like this:

• AddTwoNumbers_Variant0
• AddTwoNumbers_Variant1

The reason for this is that the numbers in the first column are the same, and generated unit test names of course need to be unique. So even if there is only a single duplicate value in that first column, test names will be generated ending with Variant[incrementing number here].

If this is the case, the VsTestScenarioOutlineExampleMatcher in Pickles will not be able to match the examples of the Scenario Outline with the unit test result. It expects the test name to end in the value of that first column. This leads to all tests of the Scenario Outline being marked as Inconclusive.

I've been looking into this and unfortunately, the .trx test results file generated by VsTest is way less useable than MsTest's in this respect, where matching is much easier because of extra information in the results file. This is how VsTest's trx file looks like. I've removed things like GUID's etc. which are not useable, which means there isn't a lot to work with:

  <Results>
      <UnitTestResult testName="AddTwoNumbers_Variant0" outcome="Passed">
      <Output>
        <StdOut>-> Using app.config
Given I have entered 1 into the calculator
-> done: StepDefinition1.GivenIHaveEnteredIntoTheCalculator(1) (0.0s)
And I have entered 2 into the calculator
-> done: StepDefinition1.GivenIHaveEnteredIntoTheCalculator(2) (0.0s)
When I press add
-> done: StepDefinition1.WhenIPressAdd() (0.0s)
Then the result should be 3 on the screen
-> done: StepDefinition1.ThenTheResultShouldBeOnTheScreen(3) (0.0s)
         </StdOut>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="AddTwoNumbers_Variant0">
      <TestCategory>
        <TestCategoryItem TestCategory="mytag" />
      </TestCategory>
      <TestMethod name="AddTwoNumbers_Variant0" />
    </UnitTest>
  </TestDefinitions>

I can't come up with a good solution for this at the moment, so I hope one of you might have a great idea. I'd be happy to build it! Looking forward to your ideas and thoughts.

[dirkrombauts]

Hi,

The usual solution is to provide an additional column in the beginning that uniquely identifies a row - ideally a small comment that says why this row is special.

Scenario Outline: Add two numbers
	Given I have entered <number1> into the calculator
	And I have entered <number2> into the calculator
	When I press add
	Then the result should be <number3> on the screen

Examples: 
	| case                 | number1 | number2 | number3 |
	| adjacent numbers     | 1       | 2       | 3       |
	| non-adjacent numbers | 1       | 3       | 4       | 

[rik-smeets]

Hi,

Thank you for your response. However, adding another column feels a bit like a workaround to me. When automating my examples, I do not want to change them just because of technical difficulties. The examples are a result of collaborative sessions, and should not be changed during automation.

However, I do not think the current VsTest test results file provides a way to resolve this issue. The only workaround is falling back to MsTest, which is however no longer maintained. Maybe I'll open an issue over at the vstest repository.

[dirkrombauts]

Hi,

I understand your feelings about this being a workaround. This is the best solution that I know of. You might want to ask this question in the SpecFlow and Cucumber communities as well, maybe they have a better solution by now.

In any case, it's not only VsTest that suffers from this problem. As far as I know, SpecFlow+ Runner (formerly known as SpecRun) also has it.

[rik-smeets]

Using a custom logger with VsTest might resolve this issue. I found this one, which should create a .trx like MsTest's. I'll check that out when I have a bit of spare time again, sorry for the delays.