Discrepancies with DataProvider and Retry of failed tests

[krmahadevan]

TestNG Version

Note: only the latest version is supported
7.7.1 (behaviour exists in 6.14.3 as well)

• I have a test method
• The test method is powered by a data provider
• The test method also has a RetryAnalyzer to perform retries
• Test method fails
• TestNG invokes the RetryAnalyser to attempt at retrying the test
• But in doing so, it ends-up invoking the data provider once again.

In short, TestNG re-invokes data provider when it deals with a failed test that is being retried, even though it does not use data computed by the re-invocation.

Expected behaviour

Data provider for a test method should be called only once, irrespective of whether a particular iteration of the test method, fails.

Actual behaviour

TestNG re-invokes data provider when it deals with a failed test that is being retried, even though it does not use data computed by the re-invocation.

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

import com.rationaleemotions.DemoTest.Commentary;
import java.util.concurrent.atomic.AtomicInteger;
import org.testng.Assert;
import org.testng.IExecutionListener;
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(Commentary.class)
public class DemoTest {

  public static final AtomicInteger dataProviderInvocationCount = new AtomicInteger(0);

  @Test(dataProvider = "dp", retryAnalyzer = TryAgain.class)
  public void testMethod(Pojo user) {
    System.out.println("Test is started for data " + user.toString());
    Assert.fail();
  }

  @DataProvider(name = "dp")
  public static Object[][] getTestData() {
    System.err.println(
        "Data Provider invocation count #" + dataProviderInvocationCount.incrementAndGet());
    return new Object[][]{
        {new Pojo().setName("John")}
    };
  }

  public static class TryAgain implements IRetryAnalyzer {

    private int counter = 1;

    @Override
    public boolean retry(ITestResult result) {
      return counter++ != 3;
    }
  }

  public static class Pojo {

    private String name;

    public Pojo() {
      System.err.println("Instantiated " + this);
    }

    public String getName() {
      return name;
    }

    public Pojo setName(String name) {
      this.name = name;
      return this;
    }
  }

  public static class Commentary implements IExecutionListener {

    @Override
    public void onExecutionStart() {}

    @Override
    public void onExecutionFinish() {
      System.err.println(
          "Data Provider was invoked " + dataProviderInvocationCount.get() + " time(s).");
    }
  }
}

Below is the output (Notice how the data provider is being invoked multiple times but the object that is being used to run the test is the same (even though a new one was created every-time the data provider was re-invoked)

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Data Provider invocation count #1
Instantiated com.rationaleemotions.DemoTest$Pojo@72cde7cc
Test is started for data com.rationaleemotions.DemoTest$Pojo@72cde7cc

Test ignored.
Data Provider invocation count #2
Instantiated com.rationaleemotions.DemoTest$Pojo@f736069
Test is started for data com.rationaleemotions.DemoTest$Pojo@72cde7cc

Test ignored.
Data Provider invocation count #3
Instantiated com.rationaleemotions.DemoTest$Pojo@4229bb3f
Test is started for data com.rationaleemotions.DemoTest$Pojo@72cde7cc

java.lang.AssertionError: null

	at org.testng.Assert.fail(Assert.java:110)
	at org.testng.Assert.fail(Assert.java:115)
	at com.rationaleemotions.DemoTest.testMethod(DemoTest.java:21)

===============================================
Default Suite
Total tests run: 3, Passes: 0, Failures: 1, Skips: 0, Retries: 2
===============================================

Data Provider was invoked 3 time(s).

Process finished with exit code 0

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.