Basic Negative Testing

In part one, we started writing a factorial function in the test-driven style. We depend on the humble “assert” statement to run our tests and we ended with this:

def factorial(n):
    return n * factorial(n-1) if n else 1
assert factorial(0) == 1
assert factorial(2) == 2
assert factorial(5) == 120

The code is correct, at least according to the tests we’ve written, but that doesn’t mean it’s bug-free.

>>> factorial(-1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "fac9.py", line 24, in factorial
    return n * factorial(n-1) if n else 1
  File "fac9.py", line 24, in factorial
    return n * factorial(n-1) if n else 1
  File "fac9.py", line 24, in factorial
    return n * factorial(n-1) if n else 1
  [Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded

Oops. Somebody passed a negative number to our factorial and overflowed the stack.

When we find a bug, since we’re writing tests first, we don’t start by fixing the bug. Instead, we start by writing a test that exposes the bug.

Here, we forgot to test for a “negative case.” “Negative testing” is a slippery concept, but it more-or-less means testing for errors.

We could argue that our current code is behaving correctly: factorials of negatives aren’t well-defined so throwing an error is appropriate. Yet this error isn’t helpful. We should throw a more informative error, so callers can understand better what they did wrong.

We can’t write this assertion like the others though: to test that a function throws an exception, we need to catch the exception:

def factorial(n):
    return n * factorial(n-1) if n else 1
...
try:
    factorial(-1)
except:
    pass

And run our test:

$ python fac.py
$

Oops. Silence… this is why we write a failing test first. If we don’t verify that the test fails, we haven’t tested the test. Here, the test isn’t testing what we intended at all; it just suppresses the error. We can easily miss this kind of mistake in our tests when we aren’t careful to make sure our test fails in the way we expect before we write the code to make it pass.

We need to test that, when our function raises an error, it raises the type of error we expect. ValueError is an appropriate built-in type here, so we’ll add that to the except line.

def factorial(n):
    return n * factorial(n-1) if n else 1
...
try:
    factorial(-1)
except ValueError as e:
    pass

Once again, we have a “red” state — test failing — and that’s a good thing.

$ python fac.py
Traceback (most recent call last):
  File "fac.py", line 28, in <module>
    factorial(-1)
  File "fac.py", line 23, in factorial
    return n * factorial(n-1) if n else 1
  File "fac.py", line 23, in factorial
    return n * factorial(n-1) if n else 1
  File "fac.py", line 23, in factorial
    return n * factorial(n-1) if n else 1
  [Previous line repeated 996 more times]
RecursionError: maximum recursion depth exceeded

So let’s fix our test in the simplest way we can.

def factorial(n):
    if n < 0: return
    return n * factorial(n-1) if n else 1
...
try:
    factorial(-1)
except ValueError as e:
    pass

Ok, silence:

$ python fac.py
$

But wait, we wanted our function to throw an error on illegal input, and it doesn’t throw anything at all. We add an assertion to verify that it doesn’t return successfully for an illegal input:

def factorial(n):
    if n < 0: return
    return n * factorial(n-1) if n else 1
...
try:
    factorial(-1)
    assert False, 'Factorial of < 0 should have raised an error'
except ValueError as e:
    pass

Now the test fails as we want:

$ python fac.py
Traceback (most recent call last):
  ...
AssertionError: factorial of < 0 should have raised an error

And we can make this last test pass.

def factorial(n):
    if n < 0:
        raise ValueError()
    return n * factorial(n-1) if n else 1
...
try:
    factorial(-1)
    assert False, 'Factorial of < 0 should have raised an error'
except ValueError as e:
    pass

ValueError by itself isn’t very informative. We can modify the except block in our test to assert that the exception has a useful message.

def factorial(n):
    if n < 0:
        raise ValueError()
    return n * factorial(n-1) if n else 1
...
try:
    factorial(-1)
    assert False, 'Factorial of < 0 should have raised an error'
except ValueError as e:
    assert str(e) == 'Factorial of negative is undefined'

Exercise for the reader: make this test pass:

$ python fac.py
...
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "fac.py", line 19, in <module>
    assert str(e) == 'Factorial of negative is undefined'
AssertionError

Next time, we’ll take our testing beyond the simple assert statement.