-
Notifications
You must be signed in to change notification settings - Fork 164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test cases of Scenario Outline not always matched with results when using VsTest #507
Comments
Hi, The usual solution is to provide an additional column in the beginning that uniquely identifies a row - ideally a small comment that says why this row is special. Scenario Outline: Add two numbers
Given I have entered <number1> into the calculator
And I have entered <number2> into the calculator
When I press add
Then the result should be <number3> on the screen
Examples:
| case | number1 | number2 | number3 |
| adjacent numbers | 1 | 2 | 3 |
| non-adjacent numbers | 1 | 3 | 4 | |
Hi, Thank you for your response. However, adding another column feels a bit like a workaround to me. When automating my examples, I do not want to change them just because of technical difficulties. The examples are a result of collaborative sessions, and should not be changed during automation. However, I do not think the current VsTest test results file provides a way to resolve this issue. The only workaround is falling back to MsTest, which is however no longer maintained. Maybe I'll open an issue over at the vstest repository. |
Hi, I understand your feelings about this being a workaround. This is the best solution that I know of. You might want to ask this question in the SpecFlow and Cucumber communities as well, maybe they have a better solution by now. In any case, it's not only VsTest that suffers from this problem. As far as I know, SpecFlow+ Runner (formerly known as SpecRun) also has it. |
Using a custom logger with VsTest might resolve this issue. I found this one, which should create a .trx like MsTest's. I'll check that out when I have a bit of spare time again, sorry for the delays. |
Implements the VsTestScenarioOutlineExampleMatcher to detect _Variant style examples. These occur when there are examples which contain characters that are illegal for method names, or when examples contain duplicate values in the first column. Fixes picklesdoc#507
Implements the VsTestScenarioOutlineExampleMatcher to detect _Variant style examples. These occur when there are examples which contain characters that are illegal for method names, or when examples contain duplicate values in the first column. Fixes picklesdoc#507
Implements the VsTestScenarioOutlineExampleMatcher to detect _Variant style examples. These occur when there are examples which contain characters that are illegal for method names, or when examples contain duplicate values in the first column. Fixes picklesdoc#507
Implements the VsTestScenarioOutlineExampleMatcher to detect _Variant style examples. These occur when there are examples which contain characters that are illegal for method names, or when examples contain duplicate values in the first column. Fixes picklesdoc#507
Implements the VsTestScenarioOutlineExampleMatcher to detect _Variant style examples. These occur when there are examples which contain characters that are illegal for method names, or when examples contain duplicate values in the first column. Fixes picklesdoc#507
Consider the following Gherkin scenario:
SpecFlow will generate the unit test names like this:
AddTwoNumbers_Variant0
AddTwoNumbers_Variant1
The reason for this is that the numbers in the first column are the same, and generated unit test names of course need to be unique. So even if there is only a single duplicate value in that first column, test names will be generated ending with
Variant[incrementing number here]
.If this is the case, the
VsTestScenarioOutlineExampleMatcher
in Pickles will not be able to match the examples of the Scenario Outline with the unit test result. It expects the test name to end in the value of that first column. This leads to all tests of the Scenario Outline being marked as Inconclusive.I've been looking into this and unfortunately, the .trx test results file generated by VsTest is way less useable than MsTest's in this respect, where matching is much easier because of extra information in the results file. This is how VsTest's trx file looks like. I've removed things like GUID's etc. which are not useable, which means there isn't a lot to work with:
I can't come up with a good solution for this at the moment, so I hope one of you might have a great idea. I'd be happy to build it! Looking forward to your ideas and thoughts.
The text was updated successfully, but these errors were encountered: