JUnitReportReporter should capture the test case output at the test case level

[JamesSassano]

TestNG Version

7.8.0

Expected behavior

The junitreports/*.xml report adds the org.testng.Reporter's test case output at the test suite level instead of the test case level. This creates ambiguity for tools which use the xml files, particularly when using the Jenkins Junit plugin which only saves the first system-out tag seen in its result. The Junit UI then renders the Standard Output for each test case with the same suite-level value. However, the Junit plugin allows for tools to "report stdout and stderr at testcase level":

• https://github.com/jenkinsci/junit-plugin/blob/adf33f3bc6569dcaf99bf066e603b7efc710f74d/src/main/java/hudson/tasks/junit/CaseResult.java#L88

The desired JUnitReportReporter output would be (some elements/attributes omitted for conciseness, some new lines added only for example clarity):

<testsuite failures="0" tests="2" name="test.Sample" errors="0" skipped="0">

  <system-out>
    <![CDATA[before class
test one
test two]]>
  </system-out>

  <testcase classname="test.Sample" name="testOne">
    <system-out>
      <![CDATA[test one]]>
    </system-out>
  </testcase> <!-- testOne -->

  <testcase classname="test.Sample" name="testTwo">
    <system-out>
      <![CDATA[test two]]>
    </system-out>
  </testcase> <!-- testTwo -->

</testsuite> <!-- test.Sample -->

Actual behavior

Current JUnitReportReporter output is generated with system-out following each testcase:

<testsuite failures="0" tests="2" name="test.Sample" errors="0" skipped="0">

  <testcase classname="test.Sample" name="testOne"/>
  <system-out>
    <![CDATA[test one]]>
  </system-out>

  <testcase classname="test.Sample" name="testTwo"/>
  <system-out>
    <![CDATA[test two]]>
  </system-out>

</testsuite> <!-- test.Sample -->

Aligning with XMLReporter

An initial task to add the Reporter output to the Junit report, #2124, mentions similarity with XMLReporter's testng-results.xml. However XMLReporters approach is to:

1. Add the full Reporter output, including all logs called during all test cases and any @Before/After configuration methods, as a child of the testng-results.
2. Add each test case's Reporter output as a child element of the test method. Note that XMLReporter additionally records before/after configuration methods as test methods.

A sample of the current XMLReporter's testng-results.xml (with some elements/attributes omitted for conciseness) to demonstrate how the new output would be aligned:

<testng-results ignored="0" total="2" passed="2" failed="0" skipped="0">
  <reporter-output>
    <line>
      <![CDATA[before class]]>
    </line>
    <line>
      <![CDATA[test one]]>
    </line>
    <line>
      <![CDATA[test two]]>
    </line>
  </reporter-output>
  <suite>
    <test>
      <class name="test.Sample">

        <test-method is-config="true" name="beforeClass">
          <reporter-output>
            <line>
              <![CDATA[before class]]>
            </line>
          </reporter-output>
        </test-method> <!-- beforeClass -->

        <test-method name="testOne">
          <reporter-output>
            <line>
              <![CDATA[test one]]>
            </line>
          </reporter-output>
        </test-method> <!-- testOne -->

        <test-method name="testTwo">
          <reporter-output>
            <line>
              <![CDATA[test two]]>
            </line>
          </reporter-output>
        </test-method> <!-- testTwo -->

      </class> <!-- test.Sample -->
    </test> <!-- test.Sample0 -->
  </suite> <!-- suite -->
</testng-results>

Junit Plugin Expected behavior

With the JUnitReportReporter update the Junit retains the stdout with the case:

<result plugin="[email protected]">
  <suites>
    <suite>
      <name>test.Sample</name>

      <stdout>
    before class
test one
test two
  </stdout>

      <cases>

        <case>
          <className>test.Sample</className>
          <testName>testOne</testName>
          <stdout>test one</stdout>
        </case>

        <case>
          <className>test.Sample</className>
          <testName>testTwo</testName>
          <stdout>test two</stdout>
        </case>

      </cases>
    </suite>
  </suites>
</result>

Junit Plugin Actual behavior

When Junit reads the current file, only one JUnitReportReporter testsuite/system-out tag is preserved as stdout in junitResult.xml:

<result plugin="[email protected]">
  <suites>
    <suite>
      <name>test.Sample</name>

      <stdout>test one</stdout>

      <cases>

        <case>
          <className>test.Sample</className>
          <testName>testOne</testName>
        </case>

        <case>
          <className>test.Sample</className>
          <testName>testTwo</testName>
        </case>

      </cases>
    </suite>
  </suites>
</result>

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

public class Sample {
  @BeforeClass
  public void beforeClass() {
    Reporter.log("before class");
  }

  @Test
  public void testOne() {
    Reporter.log("test one");
  }

  @Test
  public void testTwo() {
    Reporter.log("test two");
  }
}

Contribution guidelines

Incase you plan to raise a pull request to fix this issue, please make sure you refer our Contributing section for detailed set of steps.