Skip to content

Commit

Permalink
New manual for tests
Browse files Browse the repository at this point in the history
  • Loading branch information
@IainNZ
IainNZ committed Sep 21, 2015
1 parent dc4f179 commit e385bbc
Showing 1 changed file with 171 additions and 108 deletions.
279 changes: 171 additions & 108 deletions doc/stdlib/test.rst
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
*****************************
Unit and Functional Testing
*****************************
**************
Unit Testing
**************

Testing Base Julia
------------------
Expand All @@ -20,64 +20,185 @@ binary install, you can run the test suite using ``Base.runtests()``.

.. module:: Base.Test

Test Framework
--------------
Basic Unit Tests
----------------

The ``Test`` module contains macros and functions related to testing.
A default handler is provided to run the tests, and a custom one can be
provided by the user by using the :func:`registerhandler` function.
The ``Base.Test`` module provides simple *unit testing* functionality.
Unit testing is a way to see if your code is correct by checking that
the results are what you expect. It can be helpful to ensure your code
still works after you make changes, and can be used when developing as
a way of specifying the behaviors your code should have when complete.

To use the default handler, the macro :func:`@test` can be used directly::
Simple unit testing can be performed with the :func:`@test` and
:func:`@test_throws` macros:

julia> using Base.Test
.. function:: @test ex

julia> @test 1 == 1
.. Docstring generated from Julia source
julia> @test 1 == 0
ERROR: test failed: 1 == 0
in error at error.jl:21
in default_handler at test.jl:19
in do_test at test.jl:39
Tests that the expression ``ex`` evaluates to ``true``\ . Returns a ``Pass`` ``Result`` if it does, a ``Fail`` ``Result`` if it is ``false``\ , and an ``Error`` ``Result`` if it could not be evaluated.

julia> @test error("This is what happens when a test fails")
ERROR: test error during error("This is what happens when a test fails")
This is what happens when a test fails
in error at error.jl:21
in anonymous at test.jl:62
in do_test at test.jl:37
.. function:: @test_throws extype ex

As seen in the examples above, failures or errors will print the abstract
syntax tree of the expression in question.
.. Docstring generated from Julia source
Another macro is provided to check if the given expression throws an exception of type ``extype``,
:func:`@test_throws`::
Tests that the expression ``ex`` throws an exception of type ``extype``\ .

julia> @test_throws ErrorException error("An error")
ErrorException("An error")
For example, suppose we want to check our new function ``foo(x)`` works
as expected::

julia> @test_throws BoundsError error("An error")
ERROR: test failed: error("An error")
in error at error.jl:21
in default_handler at test.jl:19
in do_test_throws at test.jl:55
julia> using Base.Test

julia> @test_throws DomainError throw(DomainError())
DomainError()
julia> foo(x) = length(x)^2
foo (generic function with 1 method)

julia> @test_throws DomainError throw(EOFError())
ERROR: test failed: throw(EOFError())
in error at error.jl:21
in default_handler at test.jl:19
in do_test_throws at test.jl:55
If the condition is true, a ``Pass`` is returned::

julia> @test foo("bar") == 9
Test Passed
Expression: foo("bar") == 9
Evaluated: 9 == 9

julia> @test foo("fizz") >= 10
Test Passed
Expression: foo("fizz") >= 10
Evaluated: 16 >= 10

If the condition is false, then a ``Fail`` is returned and an
exception is thrown::

julia> @test foo("f") == 20
Test Failed
Expression: foo("f") == 20
Evaluated: 1 == 20
ERROR: There was an error during testing
in record at test.jl:268
in do_test at test.jl:191

If the condition could not be evaluated because an exception was thrown,
which occurs in this case because :func:`length` is not defined for
symbols, an ``Error`` object is returned and an exception is thrown::

julia> @test foo(:cat) == 1
Error During Test
Test threw an exception of type MethodError
Expression: foo(:cat) == 1
MethodError: `length` has no method matching length(::Symbol)
in foo at none:1
in anonymous at test.jl:159
in do_test at test.jl:180
ERROR: There was an error during testing
in record at test.jl:268
in do_test at test.jl:191

If we expect that evaluating an expression *should* throw an exception,
then we can use :func:`@test_throws` to check this occurs::

julia> @test_throws MethodError foo(:cat)
Test Passed
Expression: foo(:cat)
Evaluated: MethodError


Working with Test Sets
----------------------

Typically a large of number of tests are used to make sure functions
work correctly over a range of inputs. In the event a test fails, the
default behavior is to throw an exception immediately. However, it is
normally preferrable to run the rest of the tests first to get a
better picture of how many errors there are in the code being tested.

The :func:`@testset` and :func:`@testloop` macros can be used to
group tests into *sets*. All the tests in a test set will be run,
and at the end of the test set a summary will be printed. If any of
the tests failed, or could not be evaluated due to an error, the
test set will then throw a ``TestSetException``.


.. function:: @testset "description" begin ... end
@testset begin ... end

.. Docstring generated from Julia source
Starts a new test set. The test results will be recorded, and if there are any ``Fail``\ s or ``Error``\ s, an exception will be thrown only at the end, along with a summary of the test results.

.. function:: @testloop "description $v" for v in (...) ... end
@testloop for x in (...), y in (...) ... end

.. Docstring generated from Julia source
As floating-point values can be imprecise, you can perform approximate
equality checks using either ``@test a ≈ b`` (where ``≈``, typed via
tab completion of ``\approx``, is the ``isapprox`` function) or use
the macros ``@test_approx_eq`` macro (which differs from ``isapprox``
in that it treats NaN values as equal and has a smaller default
tolerance) or ``@test_approx_eq_eps`` (which takes an extra argument
indicating the relative tolerance)::
Starts a new test set for each iteration of the loop. The description string accepts interpolation from the loop indices. If no description is provided, one is constructed based on the variables.

We can put our tests for the ``foo(x)`` function in a test set::

julia> @testset "Foo Tests" begin
@test foo("a") == 1
@test foo("ab") == 4
@test foo("abc") == 9
end
Test Summary: | Pass Total
Foo Tests | 3 3

Test sets can all also be nested::

julia> @testset "Foo Tests" begin
@testset "Animals" begin
@test foo("cat") == 9
@test foo("dog") == foo("cat")
end
@testloop "Arrays $i" for i in 1:3
@test foo(zeros(i)) == i^2
@test foo(ones(i)) == i^2
end
end
Test Summary: | Pass Total
Foo Tests | 8 8

In the event that a nested test set has no failures, as happened here,
it will be hidden in the summary. If we do have a test failure, only
the details for the failed test sets will be shown::

julia> @testset "Foo Tests" begin
@testset "Animals" begin
@testset "Felines" begin
@test foo("cat") == 9
end
@testset "Canines" begin
@test foo("dog") == 9
end
end
@testset "Arrays" begin
@test foo(zeros(2)) == 4
@test foo(ones(4)) == 15
end
end

Arrays: Test Failed
Expression: foo(ones(4)) == 15
Evaluated: 16 == 15
in record at test.jl:297
in do_test at test.jl:191
Test Summary: | Pass Fail Total
Foo Tests | 3 1 4
Animals | 2 2
Arrays | 1 1 2
ERROR: Some tests did not pass: 3 passed, 1 failed, 0 errored.
in finish at test.jl:362


Other Test Macros
-----------------

As calculations on floating-point values can be imprecise, you can
perform approximate equality checks using either ``@test a ≈ b``
(where ``≈``, typed via tab completion of ``\approx``,
is the :func:`isapprox` function) or use :func:`isapprox` directly.

An alternative is the ``@test_approx_eq`` macro (which differs from
``isapprox`` in that it treats NaN values as equal and has a smaller
default tolerance) or ``@test_approx_eq_eps`` (which takes an extra
argument indicating the relative tolerance)::

julia> @test 1 ≈ 0.999999999

Expand Down Expand Up @@ -107,58 +228,9 @@ indicating the relative tolerance)::
in error at error.jl:22
in test_approx_eq at test.jl:68

Handlers
--------

A handler is a function defined for three kinds of arguments: ``Success``, ``Failure``, ``Error``::

# An example definition of a test handler
test_handler(r::Success) = nothing
test_handler(r::Failure) = error("test failed: $(r.expr)")
test_handler(r::Error) = rethrow(r)

A different handler can be used for a block (with :func:`with_handler`)::

julia> using Base.Test

julia> custom_handler(r::Test.Success) = println("Success on $(r.expr)")
custom_handler (generic function with 1 method)

julia> custom_handler(r::Test.Failure) = error("Error on custom handler: $(r.expr)")
custom_handler (generic function with 2 methods)

julia> custom_handler(r::Test.Error) = rethrow(r)
custom_handler (generic function with 3 methods)

julia> Test.with_handler(custom_handler) do
@test 1 == 1
@test 1 != 1
end
Success on :((1==1))
ERROR: Error on custom handler: :((1!=1))
in error at error.jl:21
in custom_handler at none:1
in do_test at test.jl:39
in anonymous at no file:3
in task_local_storage at task.jl:28
in with_handler at test.jl:24

The ``Success`` and ``Failure`` types include an additonal field, ``resultexpr``, which is a partially evaluated expression. For example, in a comparison it will contain an expression with the left and right sides evaluated.

Macros
------

.. function:: @test(ex)

.. Docstring generated from Julia source
Test the expression ``ex`` and calls the current handler to handle the result.

.. function:: @test_throws(extype, ex)

.. Docstring generated from Julia source
Test that the expression ``ex`` throws an exception of type ``extype`` and calls the current handler to handle the result.
Note that these macros will fail immediately, and are not compatible
with :func:`@testset`, so using `@test isapprox` is encouraged when
writing new tests.

.. function:: @test_approx_eq(a, b)

Expand All @@ -172,12 +244,3 @@ Macros
Test two floating point numbers ``a`` and ``b`` for equality taking in account a margin of tolerance given by ``tol``\ .

Functions
---------

.. function:: with_handler(f, handler)

.. Docstring generated from Julia source
Run the function ``f`` using the ``handler`` as the handler.

0 comments on commit e385bbc

Please sign in to comment.