Testing in Python

Anyone who has written more than a few lines of code can agree that testing is one of the most important parts of debugging (up there with documenting). Python has been designed for easy documentation and testing, which makes it quite easy to use and powerful.

The majority of documentation in Python is in two forms -- docstrings and reStructuredText (reST). Docstrings allow easy in-line code narration and, as explained below, testing, while reST is for more comprehensive documents. These are both easy to use systems that you have probably seen if you are reading this document, so I won't go into details here. If you haven't seen one or more of these technologies, I suggest reading up on them at some point (but they are not needed for this introduction).

Testing in Python, just like documentation, is split into two camps -- doctests and unit tests. Doctests are easy to write, simple tests that are essentially copy and pasted from the Python interpreter. Here is a simple example, which is embedded as a code literal block in this reST document:

>>> the_world_is_flat = 1
>>> if the_world_is_flat:
...     print "Be careful not to fall off!"
...
Be careful not to fall off!
>>>

This was copy and pasted from my interactive python session. If you want to run it, just invoke the following at the interpreter:

>>> import doctest
>>> doctest.testfile('py.test.rst') #doctest: +ELLIPSIS,+SKIP
(0,...)
>>>

This indicates that this file (py.test.rst) has no errors in it. Had there been a problem, it would have been reported.

This tutorial, while teaching the concepts of testing, is mostly about a certain unit testing framework, py.test. The standard testing framework for Python, unittest, was taken from Java, which stole it from Smalltalk. Consequently, it is not very pythonic -- it is almost entirely superclassed object based and makes testing quite a pain. Py.test, on the other hand, is easy to use and greatly simplifies testing.

Setup

Py.test is a very easy to set up, easy to use testing framework. It prides itself in having almost no API -- everything is controlled through name mangling and other tricks. In order to install py.test on Mac OS X or GNU/Linux, go over to codespeak.net and download the most recent version of py at http://codespeak.net/py/dist/download.html. Uncompress it and run the following commands in the resulting directory:

$ python setup.py build
$ sudo python setup.py install

Now there should be a working 'py.test' command that can be run. I imagine that the process is similar on Windows, but check the py download page for details.

Simple test

Let's try a painfully simple example of testing with py.test to get started. Suppose we have the following Python code:

def test_simple():
    assert "42 is the answer" == str(42)+" "+"is the answer"

The assert statement just makes sure that the statement following it succeeds, and the test 'fails' if the statement fails (usually a statement fails if its value is False or there is an error).

That's it! To run this test with py.test, save it as 'simple_test.py' and run:

$ py.test simple_test.py
=================================================================== test process starts ===================================================================
executable:   /opt/local/bin/python2.5  (2.5.1-final-0)
using py lib: /opt/local/lib/python2.5/site-packages/py <rev unknown>

simple_test.py[1] .

======================================================== tests finished: 1 passed in 0.02 seconds =========================================================

As you can see, the test is recognized and run. This is because it starts with 'test_' -- anything else (except some special classes, see below) is ignored (but can still be used in tests, so import statements work for example). Here is a more complicated set of tests so run in 'simple_test.py':

def test_simple():
    assert "42 is the answer" == str(42)+" "+"is the answer"

def test_multiply():
    assert 42 == 6*7

def test_ord():
    assert ord('a')+1 == ord('b')

This can be run using exactly the same command as above:

$ py.test simple_test.py
=================================================================== test process starts ===================================================================
executable:   /opt/local/bin/python2.5  (2.5.1-final-0)
using py lib: /opt/local/lib/python2.5/site-packages/py <rev unknown>

simple_test.py[3] ...

======================================================== tests finished: 3 passed in 0.01 seconds =========================================================

One more feature of py.test that is really useful is the ability to run all the tests in a subdirectory. It will run py.test on any file that starts with 'test_' or ends with '_test' and executes the result. This means that a module with arbitrary subdirectory structure, or even an entire application, can be tested with a single command. Just run 'py.test' without arguments.

After this section, there will only be clips of Python code -- just stick them in a 'test_*.py' file and run 'py.test' to run them.

Exceptions

Suppose that a test is designed to make sure a correct exception is raised. If the exception is correctly raised, then the assert will fail and the test will. This is incorrect behavior. To fix this, there are some special test forms provided by py.test -- py.test.raises(Error, function, args) and py.test.raises(Error,"function(args)"). A very simple case is division by zero:

import py.test
def test_div_zero():
    py.test.raises(ZeroDivisionError,"1/0")

There is a similar function, py.test.deprecated_call(function, args), which tests to see if a DeprecationWarning works.

Generators

Py.test can run tests that are written like Python generators, e.g. with yield. These are most easily used for testing a function with many different inputs and validating all of them. Here is an example:

def test_multi():
    for v in range(20):
        yield pos, v

def pos(x):
    assert x >= 0

This makes sure that every number from 0 to 19 is greater than or equal to 0.

Test Selection

Suppose there is a class that you are in the process of writing. You have filled in some of the tests, but not all of them work. It is a big class, so sifting through all the broken tests is not easy. Wouldn't it be nice if py.test allowed you to test a subset of the tests? In fact, it does. Pass it '-k <prefix>' as an argument and it will only run the tests that have that prefix. This includes directories, file names, etc in the multi-file version.

Migrating versions of Python

With the pending release of Python 3.0, the ability to ensure tests on multiple versions of Python becomes very important. With a simple '--exec=<command>' argument, py.test will run whatever version of Python (or any other executable, for that matter) is specified. Testing on Python 2.6 is therefore '--exec=/path/to/python/bin/python2.6' and testing on Python 3.0 is as easy as '--exec=/path/to/python/bin/python3.0'.

Debugging by Printing

Often times, it is useful to have failed cases produce debugging output. In py.test, this is accomplished very easily with the 'print' statement -- anything printed during a failed test is automatically printed after the test in a section called 'recorded stdout'. With the '--nocapture' command line argument, this doesn't happen -- instead, the text is printed wherever it is.

Test Classes

Functions are not the only way to test code in py.test. Another method is to use classes. These provide powerful frameworks for testing in contained packages. For example, it is much easier to group tests if they are in a class. The class model is a much more effective way of testing than just having a bunch of functions. Here is an example test class:

class TestAllMath:
"""Test all math functions that work on ints."""
    def test_add(self):
        assert 1+1 == 2
    def test_sub(self):
        assert 1-1 == 0

These tests will be run just like all the functions. Any class that starts with 'Test' is considered a valid name, and there is no particular class that needs to be a parent of test classes. This is very powerful -- it means that tests can be performed just by making a subclass of the application class. Here is an example of that:

class Application(object):
    """A random application class."""
    def test_working(self):
        assert isinstance(self,object)
    def test_not_working(self):
        assert not isinstance(self,object)

class TestApplication(Application):
    def test_runs(self):
        assert isinstance(self,Application)
    def test_no_runs(self):
        assert not isinstance(self,Application)
    class TestSubroutines:
        x = 1
        def test_x(self):
            assert self.x==1

When run with the '-v' option to show which tests are run, this shows many things:

1. The tests run all their 'test_*' methods.
2. Test* classes can have Test* attributes which are run as well.
3. Application methods that begin with 'test_' are run as well!

The last one is particularly interesting -- test methods can be inserted directly into classes to be tested, then all that needs to be done to make testing possible is a subclass beginning with 'Test'.

Disabling a test

Another advantage of class-based testing is that py.test allows a disabled attribute, which is a boolean which, when True, makes py.test not run the tests.

Dealing with setup

Often times, test benches need some setup and ending code to make instances of classes, remove them once the tests are done, etc. These are provided by two functions: 1) setup_module(module) is used to set up anything needed to run a test in the module. This is usually defined at the top level of a file. It is run at least once, but can be run any number of times. 2) teardown_module(module) is used to clean up anything from setup_module. It is run once for each setup_module call.

In addition, there are four similar methods for class based testing:

1. setup_class(class) is used to make classes ready for being tested. This is probably useful for subclassing application classes, since it allows attributes to be set to correct values and readies the class for testing.
2. teardown_class(class) cleans up a class.
3. setup_method(method) is used around each method to ready the class and any other state for the tests.
4. teardown_method(method) cleans up after a method.

This is probably much easier to see in an example:

def setup_module(module):
    """A simple example of test state."""
    global x
    x = 4

def teardown_module(module):
    global x
    del(x)

def test_x_value():
    assert x == 4

Doctest integration

Doctests can easily be integrated into py.test, which is really useful and somewhat unique. Any test_*.txt or *_test.txt file will automatically have any doctests inside be run by py.test.

Formal test discovery

Now that all the testing features of py.test have been more or less covered, it is time to understand exactly how py.test finds tests. It is broken down into two steps: 1) py.test, when run on a directory, runs through that directory and any subdirectories and collects all .txt and .py files that begin with 'test_' or end with '_test'. This is the list of files it tests. 2) Then, all the doctests (in .txt files) are run while all 'test_' functions and 'test_' methods of 'Test' prefixed classes in the .py files are run. In addition, 'setup_' and 'teardown_' functions and methods are run at the appropriate times, as described above.

Distributed Testing

If there are a lot of tests to be run (or a few tests that are very, very computationally intensive), it is faster to run the tests on multiple machines in parallel and collect the results. This is supported in py.test through SSH and py.execnet, and it is very easy to set up. The only requirement is that the remote and local machines are Unix-like -- I will be using a Mac OS X and GNU/Linux machine for this example. Another important thing to mention about distributed testing is that tests are run in a random order, so do not expect one test to be run before or after another -- if necessary, use the setup and teardown functions above. And of course, the same version of Python, along with py.test, must be installed on every computer in the cluster.

dist_hosts = ['localhost','user@server:/tmp/dir/for/files']
dist_rsync_roots = ['.']
dist_remotepython = 'python2.5'
dist_nicelevel = 10
dist_boxed = False
dist_maxwait = 100
dist_taskspernode = 10

Sources

The idea of the content comes from codespeak.net/py/dist/test.html#starting-point-py-test-command-line-tool and trial and error. The formatting and layout comes from the nose tutorial at ivory.idyll.org/articles/nose-intro.html.