Summary: This project documents the common use of JUnit-style XML files by testing and CI tools, including full examples & common conventions. The goal of this project is to be the reference guide for JUnit-style XML files.
The JUnit XML file format has been made popular by the JUnit project1 and has since become the de facto standard format to exchange test results between tools. JUnit-style XML files are generated by countless testing frameworks and are processed by CI tools, build servers and test management software to read and display test results.
There is no official specification for the JUnit XML file format and various tools generate and support different flavors of this format. The goal of this project here is to document a common set of elements, attributes and conventions supported by many tools2. If you are building software that produces or consumes JUnit-style XML files, this project might be a good starting point to learn about the format.
- Basic JUnit XML structure
- Complete JUnit XML example
- Properties for suites & cases
- Attachments, screenshots and artifacts
- Common properties and conventions
- About this project
The following example shows the basic structure of a JUnit-style XML file. In this example we have two root test suites (think classes/folders), one sub test suite, and a couple of test cases (test results) in each suite.
A test case is considered successful (passed) unless there is another result element underneath it. Most tools support the result elements skipped, failure and error.
This basic example is not complete as it leaves out various attributes and additional elements supported by most tools. See the next section for a full example.
Open example XML file (junit-basic.xml) →
<?xml version="1.0" encoding="UTF-8"?>
<testsuites time="15.682687">
<testsuite name="Tests.Registration" time="6.605871">
<testcase name="testCase1" classname="Tests.Registration" time="2.113871" />
<testcase name="testCase2" classname="Tests.Registration" time="1.051" />
<testcase name="testCase3" classname="Tests.Registration" time="3.441" />
</testsuite>
<testsuite name="Tests.Authentication" time="9.076816">
<testsuite name="Tests.Authentication.Login" time="4.356">
<testcase name="testCase4" classname="Tests.Authentication.Login" time="2.244" />
<testcase name="testCase5" classname="Tests.Authentication.Login" time="0.781" />
<testcase name="testCase6" classname="Tests.Authentication.Login" time="1.331" />
</testsuite>
<testcase name="testCase7" classname="Tests.Authentication" time="2.508" />
<testcase name="testCase8" classname="Tests.Authentication" time="1.230816" />
<testcase name="testCase9" classname="Tests.Authentication" time="0.982">
<failure message="Assertion error message" type="AssertionError">
<!-- Call stack printed here -->
</failure>
</testcase>
</testsuite>
</testsuites>The following example lists all commonly supported elements, attributes and typical values used by tools that generate or read JUnit-style XML files. Not all elements are supported by all tools, and not all tools follow and understand the same conventions.
For example, test cases properties are not supported by all tools yet. Likewise, not all tools have options to output attachments. And different tools also expect or use different conventions for names and values.
That said, if you are building tools that work with JUnit-style XML files, it can be a good idea to try to support most or all of these common elements and attributes.
Open example XML file (junit-complete.xml) →
<?xml version="1.0" encoding="UTF-8"?>
<!-- <testsuites> Usually the root element of a JUnit XML file. Some tools leave out
the <testsuites> element if there is only a single top-level <testsuite> element (which
is then used as the root element).
name Name of the entire test run
tests Total number of tests in this file
failures Total number of failed tests in this file
errors Total number of errored tests in this file
skipped Total number of skipped tests in this file
assertions Total number of assertions for all tests in this file
time Aggregated time of all tests in this file in seconds
timestamp Date and time of when the test run was executed (in ISO 8601 format)
-->
<testsuites name="Test run" tests="8" failures="1" errors="1" skipped="1"
assertions="20" time="16.082687" timestamp="2021-04-02T15:48:23">
<!-- <testsuite> A test suite usually represents a class, folder or group of tests.
There can be many test suites in an XML file, and there can be test suites under other
test suites.
name Name of the test suite (e.g. class name or folder name)
tests Total number of tests in this suite
failures Total number of failed tests in this suite
errors Total number of errored tests in this suite
skipped Total number of skipped tests in this suite
assertions Total number of assertions for all tests in this suite
time Aggregated time of all tests in this file in seconds
timestamp Date and time of when the test suite was executed (in ISO 8601 format)
file Source code file of this test suite
-->
<testsuite name="Tests.Registration" tests="8" failures="1" errors="1" skipped="1"
assertions="20" time="16.082687" timestamp="2021-04-02T15:48:23"
file="tests/registration.code">
<!-- <properties> Test suites (and test cases, see below) can have additional
properties such as environment variables or version numbers. -->
<properties>
<!-- <property> Each property has a name and value. Some tools also support
properties with text values instead of value attributes. -->
<property name="version" value="1.774" />
<property name="commit" value="ef7bebf" />
<property name="browser" value="Google Chrome" />
<property name="ci" value="https://github.com/actions/runs/1234" />
<property name="config">
Config line #1
Config line #2
Config line #3
</property>
</properties>
<!-- <system-out> Optionally data written to standard out for the suite.
Also supported on a test case level, see below. -->
<system-out>Data written to standard out.</system-out>
<!-- <system-err> Optionally data written to standard error for the suite.
Also supported on a test case level, see below. -->
<system-err>Data written to standard error.</system-err>
<!-- <testcase> There are one or more test cases in a test suite. A test passed
if there isn't an additional result element (skipped, failure, error).
name The name of this test case, often the method name
classname The name of the parent class/folder, often the same as the suite's name
assertions Number of assertions checked during test case execution
time Execution time of the test in seconds
file Source code file of this test case
line Source code line number of the start of this test case
-->
<testcase name="testCase1" classname="Tests.Registration" assertions="2"
time="2.436" file="tests/registration.code" line="24" />
<testcase name="testCase2" classname="Tests.Registration" assertions="6"
time="1.534" file="tests/registration.code" line="62" />
<testcase name="testCase3" classname="Tests.Registration" assertions="3"
time="0.822" file="tests/registration.code" line="102" />
<!-- Example of a test case that was skipped -->
<testcase name="testCase4" classname="Tests.Registration" assertions="0"
time="0" file="tests/registration.code" line="164">
<!-- <skipped> Indicates that the test was not executed. Can have an optional
message describing why the test was skipped. -->
<skipped message="Test was skipped." />
</testcase>
<!-- Example of a test case that failed. -->
<testcase name="testCase5" classname="Tests.Registration" assertions="2"
time="2.902412" file="tests/registration.code" line="202">
<!-- <failure> The test failed because one of the assertions/checks failed.
Can have a message and failure type, often the assertion type or class. The text
content of the element often includes the failure description or stack trace. -->
<failure message="Expected value did not match." type="AssertionError">
<!-- Failure description or stack trace -->
</failure>
</testcase>
<!-- Example of a test case that had errors. -->
<testcase name="testCase6" classname="Tests.Registration" assertions="0"
time="3.819" file="tests/registration.code" line="235">
<!-- <error> The test had an unexpected error during execution. Can have a
message and error type, often the exception type or class. The text
content of the element often includes the error description or stack trace. -->
<error message="Division by zero." type="ArithmeticError">
<!-- Error description or stack trace -->
</error>
</testcase>
<!-- Example of a test case with outputs. -->
<testcase name="testCase7" classname="Tests.Registration" assertions="3"
time="2.944" file="tests/registration.code" line="287">
<!-- <system-out> Optional data written to standard out for the test case. -->
<system-out>Data written to standard out.</system-out>
<!-- <system-err> Optional data written to standard error for the test case. -->
<system-err>Data written to standard error.</system-err>
</testcase>
<!-- Example of a test case with properties -->
<testcase name="testCase8" classname="Tests.Registration" assertions="4"
time="1.625275" file="tests/registration.code" line="302">
<!-- <properties> Some tools also support properties for test cases. -->
<properties>
<property name="priority" value="high" />
<property name="language" value="english" />
<property name="author" value="Adrian" />
<property name="attachment" value="screenshots/dashboard.png" />
<property name="attachment" value="screenshots/users.png" />
<property name="description">
This text describes the purpose of this test case and provides
an overview of what the test does and how it works.
</property>
</properties>
</testcase>
</testsuite>
</testsuites>Properties can be included for suites and cases to record additional information, such as version numbers, environment variables or links to CI pipelines. Properties are name / value pairs. Some tools support multi-line properties by including the property value as the element's text content.
Open example XML file (testcase-properties.xml) →
Open example XML file (testcase-output.xml) →
<testsuite name="Tests.Registration" time="7.005871">
<properties>
<property name="version" value="1.774" />
<property name="commit" value="ef7bebf" />
<property name="browser" value="Google Chrome" />
<property name="ci" value="https://github.com/actions/runs/1234" />
<property name="config"> <!-- Some tools support properties with text content -->
Config line #1
Config line #2
Config line #3
</property>
</properties>
<!-- Test cases here -->
</testsuite>Originally, properties were only supported for test suite elements, not cases. However, many tools support properties for test cases now, including test frameworks such as pytest and Playwright, or test management tools such as Testmo and many others. Often it can be useful to include additional details for test cases such as custom fields, test data or attachments.
<testcase name="testCase3" classname="Tests.Registration" time="3.441">
<properties>
<property name="priority" value="high" />
<property name="language" value="english" />
<property name="author" value="Adrian" />
<property name="attachment" value="screenshots/dashboard.png" />
<property name="attachment" value="screenshots/users.png" />
<!-- Optional support for properties with text values -->
<property name="description">
This text describes the purpose of this test case and provides
an overview of what the test does and how it works.
</property>
</properties>
<!-- Optional output or error/failure/skipped -->
</testcase>There can be multiple properties with the same name (e.g. multiple properties with the name attachment). Tools that read XML files should support this. However, many test frameworks have APIs that only support one property per name. In this case you should choose unique names for properties (e.g. attachment1 and attachment2, see below).
Many test frameworks don't support properties for test cases yet. Another option to include properties in XML files is to add them to the test output (similar to referencing attachments in the output). The advantage of this is that developers and testers can output properties even if their testing tool doesn't support test case properties.
<testcase name="testCase3" classname="Tests.Registration" time="3.441">
<system-out>
Output line #1
Output line #2
[[PROPERTY|author=Adrian]]
[[PROPERTY|language=english]]
[[PROPERTY|browser-log]]
Log line #1
Log line #2
Log line #3
[[/PROPERTY]]
</system-out>
</testcase>Each property is listed on its own line without any additional text. Each property consists of a name / value pair. Optionally, multi-line properties just specify the name without a value in the opening tag and end with a closing tag on its own line.
Tests often generate or use additional files as part of a test run, such as log files, screenshots or build artifacts. Sometimes it can be useful to reference or include such files in the XML file. For example, for browser and UI tests it can be useful to include one or more screenshots to better understand why a test failed. Many testing tools support one or more of the following options for attachments.
The simplest way to reference files is to list local file paths as properties. This can be included for test suites and test cases. Different tools expect different property names for attachments, but many tools simply expect the properties to be named attachment (or alternatively attachment1 etc.).
<testcase name="testCase3" classname="Tests.Registration" time="3.441">
<properties>
<property name="attachment" value="screenshots/dashboard.png" />
<property name="attachment" value="screenshots/users.png" />
</properties>
</testcase>Files might also be uploaded to and stored with CI services or on build servers. For such scenarios it can be useful to reference files as URLs.
<testcase name="testCase3" classname="Tests.Registration" time="3.441">
<properties>
<property name="attachment1" value="https://build-server/runs/45112/screenshot1.png" />
<property name="attachment2" value="https://build-server/runs/45112/screenshot2.png" />
</properties>
</testcase>For smaller files and to make it easier to record and transfer files together with an XML file, attachments can also be included inline as data URIs. Filenames can optionally be included with the non-standard filename attribute. Data should be encoded with base64.
<testcase name="testCase3" classname="Tests.Registration" time="3.441">
<properties>
<property name="attachment1" value="data:image/png;filename=dot.png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAATXL0Y4OHwAAAABJRU5ErkJggg==" />
<property name="attachment2" value="data:text/html;base64,SGVsbG8sIFdvcmxkIQ==" />
</properties>
</testcase>Some tools support another way to reference files from the test output. This convention was first introduced by the Jenkins JUnit Attachments plugin. This format is often used with tools that don't support test case properties. Each attachment is listed on a separate line (without any additional text) as part of the test's output (system-out or system-err) using the following format.
<testcase name="testCase3" classname="Tests.Registration" time="3.441">
<system-out>
Output line #1
Output line #2
[[ATTACHMENT|screenshots/dashboard.png]]
[[ATTACHMENT|screenshots/users.png]]
</system-out>
</testcase>Test frameworks and testing tools can generally write and read properties with any name and value. There aren't any standards on how to name properties or how to format your values. However, various testing tools that read JUnit-style XML files expect certain property names and values. This section lists examples for commonly used property names and how tools format property values.
Open example XML file (conventions.xml) →
Name: attachment*
Files attached to suites or cases, such as log files, screenshots, build artifacts or configurations.
<!-- Either use multiple properties named "attachment" -->
<property name="attachment" value="screenshots/failure1.png" />
<property name="attachment" value="screenshots/failure2.png" />
<!-- Or append a unique index if your test framework requires this -->
<property name="attachment1" value="browser/console.log" />
<property name="attachment2" value="browser/config.json" />Name: step*
Steps can be useful to document the exact steps a test case performs. Tools can render
all steps with optional statuses and sub fields.
<!-- Either use multiple properties named "step" -->
<property name="step" value="This is a test step." />
<property name="step" value="Tools will automatically index all steps." />
<!-- Or append a unique index if your test framework requires this -->
<property name="step1" value="You can also use unique step property names if needed." />
<property name="step2" value="And this is another test step." />
<!-- Some tools support properties with text content -->
<property name="step">
This is a more complex test step with multiple lines
of text using a property with a text value.
</property>
<!-- You can optionally also include a step result as part of the property name.
Check your CI or test management tool if it supports statuses. -->
<property name="step[passed]" value="This is a test step that passed." />
<property name="step[failure]" value="And this step had a failure." />
<property name="step[skipped]" value="This step wasn't executed." />
<property name="step[error]" value="And this step had an error." />
<!-- Some tools support custom HTML tags to document step sub fields. Check
your CI or test management tool to learn about HTML support. -->
<property name="step[passed]"><![CDATA[
<test-step-subfield name="Step">
This line describes this step. You can also use <strong>HTML</strong>.
</test-step-subfield>
<test-step-subfield name="Expected">
You can optionally include more sub fields, e.g. for the expected results.
</test-step-subfield>
]]></property>Sometimes it can be useful to use more complex types for property values, such as HTML. Or tools support rendering specific property values in a special way. For example, URLs could be rendered as clickable links.
To avoid tools having to guess the type of a property value by its content, some tools support type hints for property values. Tools that generate JUnit-style XML files (e.g. test automation frameworks) do not need to provide special support for this, as type hints are just provided as part of property names, using the format type:name.
There is no standard of types that tools support, so refer to your CI or test management tool's documentation to see if and which types might be supported. Here are some examples for types that tools might support:
<!-- Some tools also support type hints as part of the property names. Check your
CI or test management tool to learn if and which type hints are supported. Type hints
are removed from the property names. -->
<!-- URLs can be rendered as links -->
<property name="url:github" value="https://github.com/org/project/blob/code.js#L200" />
<!-- Console values are rendered as monospaced code block -->
<property name="console:log">
Log line #1
Log line #2
Log line #3
</property>
<!-- Some tools might support a limited subset of HTML for rich content -->
<property name="html:output">
<![CDATA[<h1>Title</h1><p>Text</p>]]>
</property>The goal of this project is to document the common use of JUnit-style XML files and describe how many tools implement support for this format. This project can be a starting point for builders of tools that generate or consume JUnit-style XML files. This file format has evolved over time and tools are adding support for test case properties and attachments as outlined in the above examples.
The information in this project has been collected by the team at Testmo. Please submit an issue if you have questions or feedback or think some additional details should be added.