Save test results in JUnit XML Standard

[TheElan]

It would be helpful to bee able to save test results in JUnit XML Standard.

It is very popular format and many tools already support it. For example GitLab can display test CI job report on merge request page.

P.S.: If this is any help, WAT managed to do it: watplugin/wat#59

[bitwes]

That's a great idea. If anyone has a better spec please comment. I'll base the rest of this post off of this one: Linux Lighting Group.

Until this gets implemented, you can make your own using a Hook script. In your hook script you can do gut.get_summary() and get all the summary object mentioned below. If you make one, share it, it'd be a great start for the feature.

XML Tag Mapping (in progress)

The addons/gut/summary.gd object should have most of the info. Some accessors might be required to make it easier to use but the data is there.

<testsuites>

Totals can be retrieved from summary.gd with get_totals(). This has counts for passing, pending, failing, tests, scripts.

Attributes:

• disabled: maps to pending tests.
• errors: could be the GUT errors. These are not included in the summary object yet but they are in GUT somewhere.
• tests: would be get_totals().tests
• time: is not in summary.gd but is in GUT somewhere.

<testsuite>

_scripts contains information about each script (accessible via get_scripts()). Each Inner Test Class is a separate entry in the array. Each element in the array maps directly to a <testsuite> tag.
Attributes:

• tests: TBD
• disabled: TBD
• errors: TBD
• failures: TBD
• hostname: TBD
• id: TBD
• package: TBD
• skipped: TBD
  *timestamp: TBD

<testcase>

• Each element in _scripts is a addons/gut/summary.gd:TestScript object.
• Each TestScript object has a dictionary keyed off of the test name. Each element in the dictionary would be a <testcase> tag. The value of each entry is a addons/gut/summary.Test object which has all the failure and pending messages.

Attributes:

• assertions: TBD
• classname: TBD
• status: TBD
• time: TBD

Other Requirements

Both the GUI and the command line will need an option that specifies the output file name. Something like junit_xml_result_path would indicate the format and the path in a single variable. This will allow for additional formats in the future if any are needed.

[bitwes]

You can export test results in the JUnit XML format specified here. You can specify a file name to export to, or kick off an export in a post-run hook.

Setting the export file

There are two settings, the file name and a flag to include an epoch timestamp in the filename. The epoch timestamp will prevent runs from overwritting the last run's file.

• Inspector
  • Set Junit Xml File to the path.
  • Check Junit Xml Timestamp if you want the timestamp to be included.
• Command Line
  • Set -gjunit_xml_file to the path.
  • Add option -gjunit_xml_timestamp to include the timestamp.
• .gutconfig.json File
  • "junit_xml_file":"user://resutls.xml"
  • "junit_xml_timestamp":false

Exporting results in post-run Hook

Here is sample code to export results in a post-run hook script. See Hooks for more information about setting up a post-run script.

Note that JunitXmlExport and gut are defined by the parent script res://addons/gut/hook_script.gd

extends  'res://addons/gut/hook_script.gd'

func run() # called automatically by Gut
	var exporter = JunitXmlExport.new()
    var filename = 'user://my_post_hook_results.xml'
	var f_result = exporter.write_file(gut, filename)
	if(f_result == OK):
		gut.p(str("Results saved to ", filename))

Example Output

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="GutTests" failures="7" tests="17">
      <testsuite name="res://test/resources/exporter_test_files/test_simple_2.gd" tests="3" failures="1" skipped="1">
            <testcase name="test_pass" assertions="1" status="pass" classname="res://test/resources/exporter_test_files/test_simple_2.gd"></testcase>
            <testcase name="test_fail" assertions="1" status="fail" classname="res://test/resources/exporter_test_files/test_simple_2.gd">
                  <failure message="failed">Cannot compare Int[1] to String["two"].
            at line 7</failure>
            </testcase>
            <testcase name="test_pending" assertions="0" status="pending" classname="res://test/resources/exporter_test_files/test_simple_2.gd">
                  <skipped message="pending">this has text</skipped>
            </testcase>
      </testsuite>
      <testsuite name="res://test/resources/exporter_test_files/test_simple.gd" tests="8" failures="4" skipped="2">
            <testcase name="test_pending_with_text" assertions="0" status="pending" classname="res://test/resources/exporter_test_files/test_simple.gd">
                  <skipped message="pending">this has text</skipped>
            </testcase>
            <testcase name="test_parameterized_passing" assertions="4" status="pass" classname="res://test/resources/exporter_test_files/test_simple.gd"></testcase>
            <testcase name="test_parameterized_failing" assertions="2" status="fail" classname="res://test/resources/exporter_test_files/test_simple.gd">
                  <failure message="failed">(call #1) [1] expected to equal [2]:
            at line 25</failure>
            </testcase>
            <testcase name="test_fail_2" assertions="1" status="fail" classname="res://test/resources/exporter_test_files/test_simple.gd">
                  <failure message="failed">Cannot compare String["two"] to Int[2].
            at line 13</failure>
            </testcase>
            <testcase name="test_pending_no_text" assertions="0" status="pending" classname="res://test/resources/exporter_test_files/test_simple.gd">
                  <skipped message="pending"></skipped>
            </testcase>
            <testcase name="test_pass_1" assertions="1" status="pass" classname="res://test/resources/exporter_test_files/test_simple.gd"></testcase>
            <testcase name="test_pass_2" assertions="1" status="pass" classname="res://test/resources/exporter_test_files/test_simple.gd"></testcase>
            <testcase name="test_fail_1" assertions="1" status="fail" classname="res://test/resources/exporter_test_files/test_simple.gd">
                  <failure message="failed">Cannot compare Int[1] to String["two"].
            at line 10</failure>
            </testcase>
      </testsuite>
      <testsuite name="res://test/resources/exporter_test_files/test_with_inner_classes.gd" tests="0" failures="0" skipped="0"></testsuite>
      <testsuite name="res://test/resources/exporter_test_files/test_with_inner_classes.gd.TestClassOne" tests="3" failures="1" skipped="1">
            <testcase name="test_pending_with_text" assertions="0" status="pending" classname="res://test/resources/exporter_test_files/test_with_inner_classes.gd.TestClassOne">
                  <skipped message="pending">this has text</skipped>
            </testcase>
            <testcase name="test_fail_1" assertions="1" status="fail" classname="res://test/resources/exporter_test_files/test_with_inner_classes.gd.TestClassOne">
                  <failure message="failed">Cannot compare Int[1] to String["two"].
            at line 11</failure>
            </testcase>
            <testcase name="test_pass_1" assertions="1" status="pass" classname="res://test/resources/exporter_test_files/test_with_inner_classes.gd.TestClassOne"></testcase>
      </testsuite>
      <testsuite name="res://test/resources/exporter_test_files/test_with_inner_classes.gd.TestClassTwo" tests="3" failures="1" skipped="1">
            <testcase name="test_pending_with_text" assertions="0" status="pending" classname="res://test/resources/exporter_test_files/test_with_inner_classes.gd.TestClassTwo">
                  <skipped message="pending">this has text</skipped>
            </testcase>
            <testcase name="test_fail_1" assertions="1" status="fail" classname="res://test/resources/exporter_test_files/test_with_inner_classes.gd.TestClassTwo">
                  <failure message="failed">Cannot compare Int[1] to String["two"].
            at line 26</failure>
            </testcase>
            <testcase name="test_pass_1" assertions="1" status="pass" classname="res://test/resources/exporter_test_files/test_with_inner_classes.gd.TestClassTwo"></testcase>
      </testsuite>
</testsuites>