Factory dataprovider parameters not displayed in testresult

[rouke-broersma]

When I was running my tests using just a dataProvider on the methods, the used parameter was shown in the test result and this would nicely display in jenkins using the TestNG result plugin. I am now using a Factory with dataProvider to initialize my test class, but can no longer see the used parameter anywhere in test results. This is not so useful for my selenium tests, as I would like to know which instance failed (ex chrome could have non-standard behaviour and fail while IE does not). Not having this information available makes debugging test failures much harder.

[juherr]

Could you share some examples on the observed result and the expected one?

[rouke-broersma]

I have made a gist containing my expected xml (containing a CDATA with the parameter used in the test) and my actual xml not containing CDATA.

I have also added the change I made in starting my test. TestBeforeFactory contains how I used to start my tests, passing a dataProvider to my testmethod. TestAfterFactory shows how I start my tests now, using a Factory constructor and passing the dataProvider to it.

Gist: https://gist.github.com/Mobrockers/30380bf42a8fa78906431a0a466b95bd

[juherr]

Looks like duplicate issue of #1020
Could you confirm?

[rouke-broersma]

It seems to be the same issue but I cannot confirm it has the same cause, because the person who submitted that ticket did not say how they are using parameters, just that they're missing from the report. In my case parameters are only missing when using a factory constructor dataprovider.

Probably duplicate, but can't be sure.

[juherr]

Ok, I keep both openned.

Maybe you'd like to try to fix the issue by yourself according to the analyze from the other issue? :)

[rouke-broersma]

I think the problem is here https://github.com/cbeust/testng/blob/285cf3d9825eb4ed6f05a35c1626c989e5aece96/src/main/java/org/testng/internal/FactoryMethod.java#L68-L108

The factory parameters are gathered and used to init the test classes, but they're never passed anywhere except to the class constructor after that so in effect the parameters are lost after class initialization. The code I would have to edit seems rather large and complex so I do not think I could add this functionality unfortunately.

[krmahadevan]

The below sample can be used to reproduce the problem

import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class FactorySample {

  private int number;

  @Factory(dataProvider = "dp")
  public FactorySample(int number) {
    this.number = number;
  }

  @Test
  public void testMethod() {
    System.err.println(number);
  }

  @DataProvider(name = "dp")
  public static Object[][] getData() {
    return new Object[][] {{1}, {2}, {3}};
  }
}

Currently the testng-results.xml looks like below

<?xml version="1.0" encoding="UTF-8"?>
<testng-results skipped="0" failed="0" ignored="0" total="3" passed="3">
  <reporter-output>
  </reporter-output>
  <suite name="Default Suite" duration-ms="32" started-at="2018-05-29T03:45:51Z" finished-at="2018-05-29T03:45:51Z">
    <groups>
    </groups>
    <test name="testbed" duration-ms="32" started-at="2018-05-29T03:45:51Z" finished-at="2018-05-29T03:45:51Z">
      <class name="com.rationaleemotions.github.issue1041.FactorySample">
        <test-method status="PASS" signature="testMethod()[pri:0, instance:com.rationaleemotions.github.issue1041.FactorySample@1060b431]" name="testMethod" duration-ms="6" started-at="2018-05-29T03:45:51Z" finished-at="2018-05-29T03:45:51Z">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testMethod -->
        <test-method status="PASS" signature="testMethod()[pri:0, instance:com.rationaleemotions.github.issue1041.FactorySample@11758f2a]" name="testMethod" duration-ms="1" started-at="2018-05-29T03:45:51Z" finished-at="2018-05-29T03:45:51Z">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testMethod -->
        <test-method status="PASS" signature="testMethod()[pri:0, instance:com.rationaleemotions.github.issue1041.FactorySample@612679d6]" name="testMethod" duration-ms="0" started-at="2018-05-29T03:45:51Z" finished-at="2018-05-29T03:45:51Z">
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testMethod -->
      </class> <!-- com.rationaleemotions.github.issue1041.FactorySample -->
    </test> <!-- testbed -->
  </suite> <!-- Default Suite -->
</testng-results>

[krmahadevan]

@juherr - I am not sure as to how do we go about fixing this.

For the above sample I can reproduce the problem, but then I think that the use-case itself is flawed, because the @Test method is not receiving any parameters. So TestNG is working as designed (Its the test class instance that is receiving the parameters).

But when I change the sample to something like this (Here both the constructor and the @Test methods are driven by data providers), then TestNG correctly displays the parameters in the testing-results.xml

import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;

public class FactorySample {

  private int number;

  @Factory(dataProvider = "dp")
  public FactorySample(int number) {
    this.number = number;
  }

  @Test(dataProvider = "dp1")
  public void testMethod(int number) {
    System.err.println(number);
  }

  @DataProvider(name = "dp1")
  public Object[][] getData1() {
    return new Object[][] {{1}, {2}, {3}};
  }

  @DataProvider(name = "dp")
  public static Object[][] getData() {
    return new Object[][] {{1}};
  }
}

<?xml version="1.0" encoding="UTF-8"?>
<testng-results skipped="0" failed="0" ignored="0" total="3" passed="3">
  <reporter-output>
  </reporter-output>
  <suite name="Default Suite" duration-ms="30" started-at="2018-05-29T03:51:40Z" finished-at="2018-05-29T03:51:40Z">
    <groups>
    </groups>
    <test name="testbed" duration-ms="30" started-at="2018-05-29T03:51:40Z" finished-at="2018-05-29T03:51:40Z">
      <class name="com.rationaleemotions.github.issue1041.FactorySample">
        <test-method status="PASS" signature="testMethod(int)[pri:0, instance:com.rationaleemotions.github.issue1041.FactorySample@eafc191]" name="testMethod" duration-ms="6" started-at="2018-05-29T03:51:40Z" data-provider="dp1" finished-at="2018-05-29T03:51:40Z">
          <params>
            <param index="0">
              <value>
                <![CDATA[1]]>
              </value>
            </param>
          </params>
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testMethod -->
        <test-method status="PASS" signature="testMethod(int)[pri:0, instance:com.rationaleemotions.github.issue1041.FactorySample@eafc191]" name="testMethod" duration-ms="0" started-at="2018-05-29T03:51:40Z" data-provider="dp1" finished-at="2018-05-29T03:51:40Z">
          <params>
            <param index="0">
              <value>
                <![CDATA[2]]>
              </value>
            </param>
          </params>
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testMethod -->
        <test-method status="PASS" signature="testMethod(int)[pri:0, instance:com.rationaleemotions.github.issue1041.FactorySample@eafc191]" name="testMethod" duration-ms="0" started-at="2018-05-29T03:51:40Z" data-provider="dp1" finished-at="2018-05-29T03:51:40Z">
          <params>
            <param index="0">
              <value>
                <![CDATA[3]]>
              </value>
            </param>
          </params>
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testMethod -->
      </class> <!-- com.rationaleemotions.github.issue1041.FactorySample -->
    </test> <!-- testbed -->
  </suite> <!-- Default Suite -->
</testng-results>

So what do you suggest we do here ?

[dagansandler]

I've just stumbled upon this issue myself.

It would have been nice to maybe have a testResult.getInstanceParameters() method (similar to the testResult.getParameters() method) or have that method accessible through some other object - maybe testResult.getTestClass().getParameters(), as the IClass interface has a getInstances() method which returns all the instances of that class

[krmahadevan]

@dagansandler
Assuming you have annotated your constructor with an @Factory annotation, if you are capturing the parameters that are being passed into the instance, and exposing them via your test project driven interface, then you should be able to get gold of them via ITestResult.getInstance() which can be casted to your interface and then retrieve the parameters. Would that not work for you ?

[dagansandler]

Indeed, that would work. However, we don't enforce such interfaces or inheritance in our test classes. We do extract this information via reflection though (although I agree, such interface would be better than reflection). In any case, I believe having that information exposed via testng listener interfaces would be much better, and may help develop more robust listeners which are not dependent on any test-specific implementations.

[juherr]

I agree the TestNG model is not well exposed by the API and could be improved. But it will be difficult to find a tradeoff between nonbreaking changes and nice api.
BTW, nothing is currently planned.

[sjethvani]

I am facing same problem while rerunning my failing tests (through testng-failed.xml) & merging results at the end to generate consolidated testng-results.xml file . My merge logic checks <test-method>'s 'name' attribute and <params> to uniquely identify test . Now as testng-results.xml doesn't show <params> information for @factory tests , I am not able to identify unique failing test , which is required for my custom merge logic .
Moreover I also see 'invocation-numbers'attribute missing in testng-failed.xml for @factory test, which makes testng run wrong indexed testdata when running testng-failed.xml file