[Feature] Allow users to define ordering for TestNG listeners

[krmahadevan]

As a user, I should be allowed to determine the order in which TestNG listeners will be invoked in my project (irrespective of whether those listeners were authored by me or if they were just getting resolved via some dependency).

TestNG currently has no way of ensuring that listeners are invoked in any specific order. The only round about approach that exists today is to build a container listener which would internally get all the listeners that a user wants to be registered within it and then it internally orchestrates execution of those listeners in a user determined fashion.

But this approach has a lot of pitfalls. Users have no control on listeners that were injected into TestNG via their classpath dependencies.

To mitigate all of this, TestNG should be able to do something like below:

1. User defines a comparator that basically takes in two listener objects (implementations of ITestNGListener)
2. Using the comparator the user determines the execution order.
3. User plugs in this comparator as a configuration (or) as a special listener. The special listener path if taken would mean that this listener can only have 1 instance per TestNG execution.
4. TestNG uses this new comparator to determine order and reverse-order for regular listeners and also for dealing with all reporters (including the TestNG default ones).
5. TestNG should NOT apply this comparator logic to special listeners (the notion of special listeners were introduced into TestNG as part of this commit and should be available in the next upcoming TestNG version (7.8.0 perhaps?)

For more context, refer to this discussion

[juherr]

Remarks:

• I think reporters should be excluded from this logic. They are not involved in the engine worflow (only read).
• What you call as specific listeners should be treated like any other listeners (except the internal ones which are now directly called) because they are just part of the default order logic.

[krmahadevan]

@juherr

What you call as specific listeners should be treated like any other listeners (except the internal ones which are now directly called) because they are just part of the default order logic.

What would be the expected behaviour when the user provided a comparator for ordering the listeners and still wanted some listeners to be excluded from the ordering and so specified them via the JVM argument ?

If we don't exclude them then we have broken that functionality.

[bobshie]

There is the other scenario, users define several listeners: AListener, BListener, CListener. maybe users want to invoke some afterconfig/afterinvocation as the reversed sequence with beforeconfig/afterinvocation.

for example: AListener.beforeconfig, BListener.beforeconfig, CListener.beforeconfig, CListener.afterconfig, BListener.afterconfig, AListener.afterconfig.

[juherr]

Could someone write a specification draft that will be used as documentation later?
It will allow us to discuss details on a concrete thing.

[krmahadevan]

Could someone write a specification draft that will be used as documentation later? It will allow us to discuss details on a concrete thing.

@juherr - I would like to pick this up and get this done. I have created a discussion #2916 Can we vet this out so that I can get started on this ?

[krmahadevan]

This is now merged and the changes should be available from v.7.10.0. Corresponding documentation is available at https://testng.org/#_ordering_listeners_in_testng

[TripleG]

Hi, @krmahadevan !

Now this implementation needs to be plugged into TestNG via the configuration parameter -listenercomparator

Could you please give me a hint how to pass this parameter from maven to TestNG?

[krmahadevan]

@TripleG Please try with something like this

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
    <!-- rest of the configuration goes here -->
        <properties>
            <property>
                <name>listenercomparator</name>
                <value>org.testng.MyComparator</value>
            </property>
        </properties>
    </configuration>
</plugin>

[jensenbrian]

@krmahadevan can you help me with an example of registering listenercomparator in a build.gradle file?
I am not having any luck with something like this: (I know this is not right...)

useTestNG() {
  listenercomparator = org.testng.MyComparator
}

Thank you

[krmahadevan]

@jensenbrian - I think the TestNGOptions has only limited capabilities when compared to what TestNG has. You may have to create your own custom test task

Please see here

[jensenbrian]

Thanks @krmahadevan
I have been working on creating a custom test task. While the code below "works" it is giving me build success/fail and not test pass/fail type of results. Maybe you can see something small I am missing?

Another idea:
Would it possible to add to TestNGOptions so it can be configured with useTestNG()?

Fail - Custom Test Task results look like this: (running in IntelliJ)

Execution failed for task ':useMyTestNG'.
> Process 'command '/Library/Java/JavaVirtualMachines/amazon-corretto-17.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1

Success - gives a message: "Test events were not received"

Thanks in advance for any help.

---

Custom Test Task:

task useMyTestNG(type: JavaExec) {
    dependsOn("testClasses")
    mainClass.set("org.testng.TestNG")

    List<String> arguments = new ArrayList<>()
    arguments.add("-listenercomparator")
    arguments.add("org.testng.MyComparator")

    // temp
    arguments.add("-testclass")
    arguments.add("org.testng.SimpleSoftAssertTest")

    args(arguments)

    classpath += java.sourceSets["test"].runtimeClasspath
}

[krmahadevan]

@jensenbrian

Would it possible to add to TestNGOptions so it can be configured with useTestNG()?

TestNGOptions is a gradle owned class. You might want to reach out to the Gradle team to get this sorted out.

[krmahadevan]

I tried the custom task and it works fine for me.

~/githome/playground/gradle_sample_project
./gradlew customTestNG

> Task :customTestNG
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Hello world
PASSED: sample.SampleTestCase.testMethod

===============================================
    Command line test
    Tests run: 1, Failures: 0, Skips: 0
===============================================


===============================================
Command line suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================


BUILD SUCCESSFUL in 1s
2 actionable tasks: 1 executed, 1 up-to-date

[jensenbrian]

@krmahadevan can you send me your whole build.gradle file?
I must be missing something really small.

My issue seems to be with the failing case (my success looks like yours):
Getting finished with non-zero exit value 1

===============================================
Command line suite
Total tests run: 1, Passes: 0, Failures: 1, Skips: 0
===============================================

> Task :useMyTestNG FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':useMyTestNG'.
> Process 'command '/bin/java'' finished with non-zero exit value 1

If I run a failing test as a test task (which has useTestNG()) without the custom javaexec task I do not get the finished with non-zero exit value 1 error.

./gradlew :test --tests "org.testng.SimpleSoftAssertTest"

I need the custom test task to be able to add the listenercomparator.

---

For more background... I'm working with the latest Selenide and the SoftAsserts do not work. I'm trying to add into my project the listenercomparator.

Thanks in advance

[krmahadevan]

Here's the gradle build file

build.gradle

plugins {
    id 'idea'
    id 'java'
}

repositories {
    mavenCentral()
}

dependencies {
    testImplementation 'org.testng:testng:7.10.2'
}

task customTestNG(type: JavaExec) {
    dependsOn("testClasses")
    mainClass.set("org.testng.TestNG")
    List<String> arguments = new ArrayList<>()
    arguments.add("-listenercomparator")
    arguments.add("org.testng.MyComparator")

    arguments.add("-testclass")
    arguments.add("sample.SampleTestCase")
    arguments.add("-verbose")
    arguments.add(2)
    args(arguments)

    classpath += java.sourceSets["test"].runtimeClasspath
}

And here's the execution logs of a test case that has one test method that fails.

logs

~/githome/playground/gradle_sample_project
./gradlew customTestNG

> Task :customTestNG FAILED
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Hello world
PASSED: sample.SampleTestCase.testMethod
FAILED: sample.SampleTestCase.failingTestMethod
java.lang.AssertionError: null
        at org.testng.Assert.fail(Assert.java:111)
        at org.testng.Assert.fail(Assert.java:116)
        at sample.SampleTestCase.failingTestMethod(SampleTestCase.java:19)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:141)
        at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:686)
        at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:230)
        at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:63)
        at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:992)
        at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:203)
        at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:154)
        at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:134)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
        at org.testng.TestRunner.privateRun(TestRunner.java:739)
        at org.testng.TestRunner.run(TestRunner.java:614)
        at org.testng.SuiteRunner.runTest(SuiteRunner.java:421)
        at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:413)
        at org.testng.SuiteRunner.privateRun(SuiteRunner.java:373)
        at org.testng.SuiteRunner.run(SuiteRunner.java:312)
        at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
        at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
        at org.testng.TestNG.runSuitesSequentially(TestNG.java:1274)
        at org.testng.TestNG.runSuitesLocally(TestNG.java:1208)
        at org.testng.TestNG.runSuites(TestNG.java:1112)
        at org.testng.TestNG.run(TestNG.java:1079)
        at org.testng.TestNG.privateMain(TestNG.java:1430)
        at org.testng.TestNG.main(TestNG.java:1394)


===============================================
    Command line test
    Tests run: 2, Failures: 1, Skips: 0
===============================================


===============================================
Command line suite
Total tests run: 2, Passes: 1, Failures: 1, Skips: 0
===============================================


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':customTestNG'.
> Process 'command '/Users/kmahadevan/.sdkman/candidates/java/11.0.20-ms/bin/java'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 2s
2 actionable tasks: 2 executed

[jensenbrian]

Ah, I see you also get finished with non-zero exit value 1. Unfortunately this does not play well with other tools like IntelliJ or TeamCity.

I think I might have to go to gradle side to see if we can add the listenercomparator to the useTesetNG.
Unless you have any other ideas?

Thank you

[krmahadevan]

@jensenbrian - Yeah I think you would need to approach the Gradle team to get this sorted out

[asolntsev]

@krmahadevan Sorry for reporting too late, but for me this feature request seems strange: it brings even more complexity to tests setup.
I already declare order of listeners when I add them to the test:

1. Why should I add even more classes (implement ListenerComparator) to specify order of listeners once again?
2. Right now, it's not possible to pass my custom ListenerComparator to Gradle build script.

From the point of end user, the problem is still not solved. :(

[jensenbrian]

While this is still an issue I agree this is cannot be fixed in Selenide or TestNG.
I have linked it to an existing gradle issue #18828
Thanks for your help and consideration

[krmahadevan]

@krmahadevan Sorry for reporting too late, but for me this feature request seems strange: it brings even more complexity to tests setup. I already declare order of listeners when I add them to the test:

1. Why should I add even more classes (implement ListenerComparator) to specify order of listeners once again?

Because listeners in TestNG are not scoped and irrespective of whether you specify them at individual class level or at the suite file level or via service loaders, they are always going to get hoisted globally. So in that context, how would TestNG determine order of listeners to be executed?

2. Right now, it's not possible to pass my custom ListenerComparator to Gradle build script.

I am not very well versed in Gradle. But it looks to me that either you would need to have Gradle expose a few more configurations to the level that maven surefire plugin currently does or you would need to build your custom TestNG task and then work with it.

From the point of end user, the problem is still not solved. :(

I kind of disagree to this because you are now referring to a shortcoming in a build tool. A Maven user would still be able to get this done. Perhaps you could chime in on gradle/gradle#18828 and upvote the issue ?

[asolntsev]

they are always going to get hoisted globally

I understand that generally, users might declare multiple listeners in multiple places.
But in practice, most users just declare a LIST of listeners in a single place. Most probably in base test class:

@Listeners({SoftAsserts.class, TextReport.class, BrowserPerTest.class, ScreenShooter.class})
abstract class BaseTest {

or in xml file, or something like this.

So, for most users, in most cases, it's OK if TestNG just reads listeners in the order of declaration.
It would be very good default behavior that serves most users without any additional complexity.

And only for "advanced" users who really want to declare listeners in different places, these is a possibility to declare custom ListenerComparator.