Precise Unit Tests with PyHamcrest

(This is based on my article on opensource.com)

Unit test suites help maintain high-quality products by signaling problems early in the development process. An effective unit test catches bugs before the code has left the developer machine, or at least in a continuous integration environment on a dedicated branch. This marks the difference between good and bad unit tests: good tests increase developer productivity by catching bugs early and making testing faster. Bad tests decrease developer productivity.

Productivity decreases when testing incidental features. The test fails when the code changes, even if it is still correct. This happens because the output is different, but in a way that is not part of the function's contract.

A good unit test, therefore, is one that helps enforce the contract to which the function is committed.

If a good unit test breaks, the contract is violated and should be either explicitly amended (by changing the documentation and tests), or fixed (by fixing the code and leaving the tests as is).

A good unit test is also strict. It does its best to ensure the output is valid. This helps it catch more bugs.

While limiting tests to enforce only the public contract is a complicated skill to learn, there are tools that can help.

One of these tools is Hamcrest, a framework for writing assertions. Originally invented for Java-based unit tests, today the Hamcrest framework supports several languages, including Python.

Hamcrest is designed to make test assertions easier to write and more precise.

def add(a, b):
    return a + b

from hamcrest import assert_that, equal_to

def test_add():
    assert_that(add(2, 2), equal_to(4))

This is a simple assertion, for simple functionality. What if we wanted to assert something more complicated?

def test_set_removal():
    my_set = {1, 2, 3, 4}
    my_set.remove(3)
    assert_that(my_set, contains_inanyorder([1, 2, 4]))
    assert_that(my_set, is_not(has_item(3)))

Note that we can succinctly assert that the result has 1, 2, and 4 in any order since sets do not guarantee order.

We also easily negate assertions with is_not. This helps us write precise assertions, which allow us to limit ourselves to enforcing public contracts of functions.

Sometimes, however, none of the built-in functionality is precisely what we need. In those cases, Hamcrest allows us to write our own matchers.

Imagine the following function:

def scale_one(a, b):
    scale = random.randint(0, 5)
    pick = random.choice([a,b])
    return scale * pick

We can confidently assert that the result divides into at least one of the inputs evenly.

A matcher inherits from hamcrest.core.base_matcher.BaseMatcher, and overrides two methods:

class DivisibleBy(hamcrest.core.base_matcher.BaseMatcher):

    def __init__(self, factor):
        self.factor = factor

    def _matches(self, item):
        return (item % self.factor) == 0

    def describe_to(self, description):
        description.append_text('number divisible by')
        description.append_text(repr(self.factor))

Writing high-quality describe_to methods is important, since this is part of the message that will show up if the test fails.

def divisible_by(num):
    return DivisibleBy(num)

By convention, we wrap matchers in a function. Sometimes this gives us a chance to further process the inputs, but in this case, no further processing is needed.

def test_scale():
    result = scale_one(3, 7)
    assert_that(result,
                any_of(divisible_by(3),
                       divisible_by(7)))

Note that we combined our divisible_by matcher with the built-in any_of matcher to ensure that we test only what the contract commits to.

While editing the article, I heard a rumor that the name "Hamcrest" was chosen as an anagram for "matches". Hrm...

>>> assert_that("matches", contains_inanyorder(*"hamcrest")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/moshez/src/devops-python/build/devops/lib/python3.6/site-packages/hamcrest/core/assert_that.py", line 43, in assert_that
    _assert_match(actual=arg1, matcher=arg2, reason=arg3)
  File "/home/moshez/src/devops-python/build/devops/lib/python3.6/site-packages/hamcrest/core/assert_that.py", line 57, in _assert_match
    raise AssertionError(description)
AssertionError:
Expected: a sequence over ['h', 'a', 'm', 'c', 'r', 'e', 's', 't'] in any order
      but: no item matches: 'r' in ['m', 'a', 't', 'c', 'h', 'e', 's']

Researching more, I found the source of the rumor: it is an anagram for "matchers".

>>> assert_that("matchers", contains_inanyorder(*"hamcrest"))
>>>

If you are not yet writing unit tests for your Python code, now is a good time to start. If you are writing unit tests for your Python code, using Hamcrest will allow you to make your assertion precise—neither more nor less than what you intend to test. This will lead to fewer false negatives when modifying code and less time spent modifying tests for working code.