Testing code

TODO: Update to pytest

TODO: update to refer to more recent discussions on testing in python

• use numpy.testing functions for numerical assessment

How should I test my code?

Tests are an opportunity to invent the interface(s) you want. Write the test for a method before you write the method: often, this helps you figure out what you would want to call it and what parameters it should take. It's OK to write the tests a few methods at a time, and to change them as your ideas about the interface change. However, you shouldn't change them once you've told other people what the interface is.

Never treat prototypes as production code

It's fine to write prototype code without tests to try things out, but when you've figured out the algorithm and interfaces you must rewrite it with tests to consider it finished. Often, this helps you decide what interfaces and functionality you actually need and what you can get rid of.

"Code a little test a little"

For production code, write a couple of tests, then a couple more tests, then a couple more methods, change some of the names or generalise some of the functionality. If you have a huge amount of code where 'all you have to do is write the tests', you're closer to 30% done than 90%. Testing vastly reduces the time spent debugging, since whatever went wrong has to be in the code you wrote since the your last run of the test suite. And remember to use Python's interactive interpreter for quick checks of syntax and ideas.

Run the test suite when you change anything. Even if a change seems trivial, it will only take a couple of seconds to run the tests and then you'll be sure. This can eliminate long and frustrating debugging sessions where the change turned out to have been made long ago, but didn't seem significant at the time.

Some unittest pointers

NOTE We are moving towards pytest throughout cogent3, these tips are for PyCogent, which included its own testing framework. So don't interpret these literally.

• Use the unittest framework with tests in a separate file for each module. Name the test file test_module_name.py. Keeping the tests separate from the code reduces the temptation to change the tests when the code doesn't work, and makes it easy to verify that a completely new implementation presents the same interface (behaves the same) as the old.

• Use evo.unit_test if you are doing anything with floating point numbers or permutations (use assertFloatEqual). Do not try to compare floating point numbers using assertEqual if you value your sanity. assertFloatEqualAbs and assertFloatEqualRel can specifically test for absolute and relative differences if the default behavior is not giving you what you want. Similarly, assertEqualItems, assertSameItems, etc. can be useful when testing permutations.

• Test the interface of each class in your code by defining at least one TestCase with the name ClassNameTests. This should contain tests for everything in the public interface.

• If the class is complicated, you may want to define additional tests with names ClassNameTests_test_type. These might subclass ClassNameTests in order to share setUp methods, etc.

• Tests of private methods should be in a separate TestCase called ClassNameTests_private. Private methods may change if you change the implementation. It is not required that test cases for private methods pass when you change things (that's why they're private, after all), though it is often useful to have these tests for debugging.

• Test all the methods in your class. You should assume that any method you haven't tested has bugs. The convention for naming tests is test_method_name. Any leading and trailing underscores on the method name can be ignored for the purposes of the test; however, all tests must start with the literal substring test for unittest to find them. If the method is particularly complex, or has several discretely different cases you need to check, use test_method_name_suffix, e.g. test_init_empty, test_init_single, test_init_wrong_type, etc. for testing __init__.

• Write good docstrings for all your test methods. When you run the test with the -v command-line switch for verbose output, the docstring for each test will be printed along with ...OK or ...FAILED on a single line. It is thus important that your docstring is short and descriptive, and makes sense in this context.

  Good docstrings:

    NumberList.var should raise ValueError on empty or 1-item list
    NumberList.var should match values from R if list has >2 items
    NumberList.__init__ should raise error on values that fail float()
    FrequencyDistribution.var should match corresponding NumberList var
  

  Bad docstrings:

    var should calculate variance           # lacks class name, not descriptive
    Check initialization of a NumberList    # doesn't say what's expected
    Tests of the NumberList initialization. # ditto
  

• Module-level functions should be tested in their own TestCase*, called* modulenameTests. Even if these functions are simple, it's important to check that they work as advertised.

• It is much more important to test several small cases that you can check by hand than a single large case that requires a calculator. Don't trust spreadsheets for numerical calculations -- use R instead!

• Make sure you test all the edge cases: what happens when the input is None, or '', or 0, or negative? What happens at values that cause a conditional to go one way or the other? Does incorrect input raise the right exceptions? Can your code accept subclasses or superclasses of the types it expects? What happens with very large input?

• To test permutations, check that the original and shuffled version are different, but that the sorted original and sorted shuffled version are the same. Make sure that you get different permutations on repeated runs and when starting from different points.

• To test random choices, figure out how many of each choice you expect in a large sample (say, 1000 or a million) using the binomial distribution or its normal approximation. Run the test several times and check that you're within, say, 3 standard deviations of the mean.

.. _Jim Fulton: http://www.python.org/pycon/dc2004/papers/4/PyCon2004DocTestUnit.pdf