-
Notifications
You must be signed in to change notification settings - Fork 9
TAP Y J Specification
TAP-Y and TAP-J are test streams. They are essentially the same except for the underlying format used, which are YAML and JSON repsectively.
The following overview will use the YAML format. Becuase the YAML format is a plan format, not using an special YAML tags, it is easy to convert to JSON to get the equivalent TAP-J format. TAP-J documents follow all the same feild rules as TAP-Y, but are represented as a stream of JSON documents, one per line, instead of YAML documents.
This document describes Revision 4 of the specification.
TAP-Y is a plain YAML stream format. Only YAML core types are used: scalar, sequence and mapping.
A YAML stream is composed a sequence of YAML documents, each divided by
a start document marker (---). Each document MUST have a type
field which designates it a suite, case, test, note or tally. Any
document MAY have an extra entry which contains an open mapping for
extraneous information.
A suite document marks the beginning of a forthcoming stream of tests,
i.e. a test suite. All TAP-Y streams MUST begin with a suite
document.
---
type: suite
start: 2011-10-10 12:12:32
count: 2
seed: 32154
rev: 4The start field marks the date and time testing began. It MUST be
an ISO-8601 formated timestamp.
The count field indicates the total number of test units forethcoming.
If the number of test units is unknown, the total can be omitted or
marked as ~ (nil). The total should only indicate the number of
test units, not any enumeration of test cases.
The seed is provided if the test runner has randomized the order of
execution unit tests. The seed can be used to reproduce the same order.
The rev field provides the version of the TAP-Y/J format that is
being used. The specification will change little, if at all, as it
become more mainstream. But just in case, having a revision field
ensures things will work if they do change by allowing consuming
apps to adjust to any future variation.
The case type indicates the start of a test case.
---
type: case
subtype: feature
label: Multiplication
level: 0The case document MAY provide a subtype which is a label for the
type of test case. For example, a test framwework that uses Gherkin
nomenclature would classify a test case as a "feature".
The case document SHOULD provide a label that is a free-form string
describing the nature of the test case.
The level field is used to notate sub-case heiararchies. By default the
value is assumed to be 0, which means the case is not a subcase. If 1,
it indicates that the case is a subcase of the previous zero-level
case, and so on for higher levels. Subcases should proceed sequentially.
If a case contains both tests and subcases, the tests must come first in the
document stream.
The test type indicates a test procedure. A unit MUST have a status with
one of five possible values: pass, fail, error, omit or todo.
Unit documents vary somewhat based on the status. But they all share
common fields.
Here is an example of a passing unit document.
---
type: test
subtype: step
status: pass
setup: foo instance
label: multiples of two
expected: 2
returned: 2
file: test/test_foo.rb
line: 45
source: ok 1, 2
snippet:
- 44: ok 0,0
- 45: ok 1,2
- 46: ok 2,4
coverage:
file: lib/foo.rb
line: 11..13
code: Foo#*
stdout: ''
stderr: ''
time: 0.01Besides the status, all test documents MUST have a label.
A test document MAY provide a setup field, which is used to describe
the setup for the unit test.
Tests SHOULD also give an expected and returned value, if relavent
to the nature of the test. For example, the most common test assertion is
equality, e.g. assert_equal(4,3), so expected would be 3 and
returned would be 4. Although desirable this can be a difficult piece
of information for some test frameworks to provide, so it is the most often
omitted.
A test SHOULD also have a file and line number for source file location.
This is the location of the test definition itself.
A test SHOULD provide the line of source code for the test.
This will be the line of code that file and line number references.
Unlike snippet lines, the source line should be stripped of whitespace.
The snippet is like source but provides surronding context. It MAY be
a verbatim string, in which case it MUST have an odd number of lines with
the source line in the center. Or, it MAY be an ordered map of verbatim
- line: source. Using an ordered map the line numbers may start
and end wherever, but they MUST be consecutive and the source line MUST
be among them.
[EXPERIMENTAL] The coverage subsection MAY be provided, in which can have three optional
fields: file, line and code. Where file specifies the source file being
targeted by the test, line specifies the line number, range of line
numbers (e.g. 1..4) or an array of such, and code specifices the
language construct being targeted. For example, for Ruby code might be Foo#bar
if the test targets the bar method of the Foo class.
The stdout and stderr fields contain any output produced while running
the test.
The time is the number of seconds that have elapsed since the
the suite start time.
If a test has a status other than pass it MUST also provide a exception
subsection which is used to describe the nature of the failure, error or
omission.
---
type: test
subtype: step
status: fail
label: multiples of two
setup: foo instance
expected: 2
returned: 1
file: test/test_foo.rb
line: 45
source: ok 1, 2
snippet:
- 44: ok 0,0
- 45: ok 1,2
- 46: ok 2,4
coverage:
file: lib/foo.rb
line: 11..13
code: Foo#*
exception:
message: |
(assertion fail) must_equal
1
2
file: test/test_foo.rb
line: 50
source: 1.must_equal == v
snippet:
- 49: v = 2
- 50: 1.must_equal == v
- 51: ''
backtrace:
- test/test_foo.rb:50
- test/test_foo.rb:45
time: 0.02The exception section MUST give the message, describing the nature
of the failure or exception. In this subsection, file and line indicate
the location in code that triggered the exception or failed assertion.
Like the originating test code, a source and code snippet SHOULD also
be provided.
It MAY also provide a system backtrace.
Q. Why supply a code snippet when the file and line are already given. Can't a test reporter just look up the code itself?A. Of course it can, but if the TAP-Y document is being consumed remotely it might not have easy access the file being tested. While this may be of rare use it none the less provides the TAP-Y consumer some view of the code with having to do additional processing.
The note type is used to interject a message between tests that
is not tied to a specific unit or case. It has only a few fields.
---
type: note
text:
This is an example note.The note document is simply used to interject any information the tester might want to know, but doesn't properly fit elsewhere in the stream. A note cna appear any where in the document stream prior to the tally.
The final and tally types are the same. The difference is only that a tally
entry is a running tally, and can technically occur anywhere in the document
stream. The final entry on the other hand incidates the end of a test suite,
which will be followed by an end-document-marker (...).
---
type : final
time : 0.03
counts:
total: 2
pass : 1
fail : 1
error: 0
omit : 0
todo : 0
...A tally/final document MUST provide a counts mapping with the total number of
tests (this MUST be same as count in the suite document if it was given)
and the totals for each test status. It SHOULD also give the time elapsed
since the suite time.
Tally documents are very rare, if used at all. They only make sense for very
large test suites as a progress report mechanism. As a rule of thumb, TAP-Y/J
consumer apps will ignore them unless a configuration option (e.g. --verbose)
is used.
As mentioned, the test stream ends when a full ellipsis (...)
appears.
As you can see TAP-Y streams provides a great deal of detail. They are not intended for the end-user, but rather to pipe to a consuming app to process into a human readable form.
The count field provides the total number of tests being executed. It SHOULD
be given in the header, if possible, and it MUST be given in the footer.
Additional data, not specifucally designated by this sepecification can
placed within an extra section of any document without worry that future
versions of the specification will come into conflict with the field name.
The field MUST be a mapping. The key namespace is a free-for-all, so use it
with that in mind.
The file field provides the name of the file in which the test is defined,
or where th test failed/errored.
The line field provides the line number of the file on which the
definition of the test begins, or is the line number of where the
test failed/errored.
A subsection used to sepcify the nature of a non-passing test.
For tests without a pass status, the message provides the explination
for the failure or error. Usually this is just the error message produced by
the underlying exception. The pass type can have the message field too,
but it will generally be ignored by TAP consumers.
The snippet field is either a verbatim string or an ordered mapping of line
number mapped to the source code for that line. While snippet is
like source it also contains extra lines of code before and after the
test line for context.
If snippet is a string it MUST consist an odd number of lines, the same
number before and after the source line in the center, unless the line occurs
at the begining or the end of the file. The number of lines before and after is
arbitrary and up to the producer, but should be the same on either side. Three
to five is generally enough.
The source field is a verbatim copy of the source code that defines the test.
This may be just the first line of the definition. In classic TAP this
is called raw_test.
The suite decument provides date/time information for when a suite of tests
began being tests. The filed MUSTbe in ISO standard format YYYY-MM-DD HH:MM:SS.
The status field designates the status of a test document. Valid values
are pass, fail, error, omit and todo.
In comparison to the classic TAP format, pass is equivalent to ok and
fail and error are akin to not ok, where fail is "not ok" with regards
to a test assertion and error is "not ok" becuase of a raised coding error.
Tests with an omit status do not need to be provided in the document stream,
so this status might not appear often in practice. But if a producer chooses to
do so this status simply means the test is purposefully being disregarded for
some reason. The exception subsection is used to clarify that reason.
On the other hand, todo means the test will be used in the future
but implementation has not been completed. It serves as reminder to developers
to write a missing test.
The footer MUST provide counts for all status categories and the total.
This is like count in the suite entry but broken down into status groups.
The tests and the footer SHOULD have the time elapsed since starting the
tests given in number of seconds.
Each document MUST have a type. Valid types are suite, tally, case,
test and note.
The suite type can only occur once, at the start of the stream. All other
types may occur repeatedly in between, although the tally type will
generally only occur at the end of a stream.
The case type marks the start of a testcase. All test (and note)
documents following it are considered a part of the case until a new case
document occurs with the same level.