TestResult lost if failure creating RetryAnalyzer

[lucianoRM]

TestNG Version: 7.9.0

Expected behavior

If the ITestListener already executed other methods for the same ITestResult, for example: onTestStart(), then the same instance should be provided for methods executed after, like: onTestFailure(). Or at least, all information added to the result should be available in every ITestListener method.

Actual behavior

Test starts running. Pre run listener hooks are (onTestStart()) are run correctly. However, a failure in the IRetryAnalyzer creation is not handled and bubbles up to the logic handling exceptions for "non-started" tests. Where in this case the test already executed.

This causes the ITestListener to receive a new instance of the ITestResult loosing all stored information.

Test case sample

ITestListener

package some.test;

import org.testng.ITestListener;
import org.testng.ITestResult;

public class EvidenceListener implements ITestListener {

    private static final String ATTRIBUTE_KEY = "attributeKey";

    @Override
    public void onTestStart(ITestResult result) {
        System.out.println("OnTestStart is executed!");
        result.setAttribute(ATTRIBUTE_KEY, "attributeValue");
    }

    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("Attribute value is: " + result.getAttribute(ATTRIBUTE_KEY));
    }
}

IRetryAnalyzer

package some.test;

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;


public class EvidenceRetryAnalyzer implements IRetryAnalyzer {

    public EvidenceRetryAnalyzer() {
        throw new RuntimeException("Failed on purpose");
    }

    @Override
    public boolean retry(ITestResult result) {
        return false;
    }
}

Evidence Test.

package some.test;

import static org.testng.AssertJUnit.fail;

import org.testng.ITestContext;
import org.testng.ITestNGMethod;
import org.testng.Reporter;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Test
@Listeners(EvidenceListener.class)
public class EvidenceTestOne {

    @BeforeSuite(alwaysRun = true)
    public void suiteSetup() {
        ITestContext context = Reporter.getCurrentTestResult().getTestContext();
        for (ITestNGMethod method : context.getAllTestMethods()) {
            method.setRetryAnalyzerClass(EvidenceRetryAnalyzer.class);
        }
    }

    @Test
    public void testOne() {
        fail();
    }

}

Running the evidence test you will see that the console prints out:

OnTestStart is executed!
Attribute value is: null

Where the attribute value retrieved should be the one stored in the onTestStart() method.

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.

[krmahadevan]

@lucianoRM

public EvidenceRetryAnalyzer() {
        throw new RuntimeException("Failed on purpose");
    }

Not sure I quite u'stand as to why would a retry analyzer be intentionally failed. Is this the correct sample ?

[lucianoRM]

Yes! This is just for testing, I wanted to simplify it a little bit.
The actual issue was because the RetryAnalyzer had a @ Guice annotation and some dependencies to other modules. If the dependency injection fails, then there is an error thrown and the behavior is the same.

[krmahadevan]

@lucianoRM - Thanks for confirming that hunch of mine. I have raised a PR to fix it. So this should be fixed in the next upcoming release.