Mocking Has A Weakness, Speccing Removes It

Mocking has several strengths, the main one being that it allows us to write tests by temporarily removing the reliance on dependencies that a piece of code might have. It does that by replacing that dependency with a mock object. The flip side of that strength is that if the code that we have mocked is changed or even broken the test might still pass!

Let’s look at an example. Assume there is a module named car.py that contains the following:

class Car(object):
    def drive(self):
        return "GOGOGOGO"
def create_and_drive():
    new_car = Car()
    return new_car.drive()

I can write simple module called use_car.py to use create_and_drive:

from car import create_and_drive
print(create_and_drive())

When run this outputs:

GOGOGOGO

Now I want to write a test for create_and_drive but let’s assume I need to mock drive:

from car import create_and_drive
from unittest import TestCase, mock
class test_car(TestCase):
    @mock.patch("car.Car.drive")
    def test_drive(self, mock_drive):
        mock_drive.return_value = "BRRRRUUUUMMMM"
        noise = create_and_drive()
        assert(noise == "BRRRRUUUUMMMM")

When run this simple test passes, but now I decide to change the drive method by adding a parameter called speed, however, I forget to update create_and_drive to add that parameter to the line return new_car.drive(), so car.py becomes:

class Car(object):
    def drive(self, speed):
        return "GOGOGOGO - {0}".format(speed)
def create_and_drive():
    new_car = Car()
    return new_car.drive()

Now when I run use_car.py I get the error:

TypeError: drive() missing 1 required positional argument: 'speed'

However, when I run the test it still passes! To emphasise this point: I changed the code and broke it, but the test that is meant to check that the code works passed! Pretty bad test! It passed because I changed the underlying function that I mocked, however, the mock silently deals with the drive method being called with or without the additional parameter. The weakness in mocking has allowed the test to pass when it shouldn’t. This doesn’t just apply to functions, whatever you mock, if you then change that thing there is a chance that any test that involves that thing will still pass.

There a couple ways to get around this:

• Write an integration test to cover any scenario that involves interacting with a dependency (and therefore needs mocking when unit testing).
• Use speccing when writing unit tests.

I recommend doing both, but I will focus on the speccing part.

Speccing

Speccing means creating a mock object that has the same api/structure as the object selected for mocking; a mock object that will error if it is used in a way that doesn’t match the spec.

There are two ways of doing this, the first is using the create_autospec function that is found in the mock module. The docstring says this about the function:

Create a mock object using another object as a spec. Attributes on the
mock will use the corresponding attribute on the `spec` object as their
spec.

Functions or methods being mocked will have their arguments checked
to check that they are called with the correct signature.

Let’s create a mock object based on the specification of the following function:

def boom(type, size, weight):
    return "BOOOOOM"

We then pass boom to create_autospec:

mock_boom = create_autospec(boom)

As you can from the above function it has three arguments. If we try and call mock_boom with just one argument like so:

mock_boom("tnt")

Then the following error is raised:

TypeError: missing a required argument: 'size'

Or if you call it with two arguments the following error is raised:

TypeError: missing a required argument: 'weight'

Or if you call it with four or more arguments:

TypeError: too many positional arguments

So as you can see if the mock object is called in any way that doesn’t meet the specification of the function boom then an error occurs.

As I mentioned, create_autospec is one way of creating a mock object from a specification, but another way is set autospec=True when using the patch decorator to mock something. To demonstrate this I will go back to the car example. Our test currently is:

@mock.patch("car.Car.drive")
def test_drive(self, mock_drive):
    mock_drive.return_value = "BRRRRUUUUMMMM"
    noise = create_and_drive()
    assert(noise == "BRRRRUUUUMMMM")

This test passes when it shouldn’t, but if we change it to:

@mock.patch("car.Car.drive", autospec=True)
def test_drive(self, mock_drive):
    mock_drive.return_value = "BRRRRUUUUMMMM"
    noise = create_and_drive()
    assert(noise == "BRRRRUUUUMMMM")

Then when run we get the following error:

TypeError: missing a required argument: 'speed'

Now the test is failing because the code is failing, which is great! This is because by setting autospec to True, the mock object mock_drive has the specification of the drive function. So when drive is called with no arguments passed in, in reality it is mock_drive that is called and because it has the specification of drive it errors accordingly.

You can also use a class as a specification. If I re-write the test to mock the Car class instead, but not do speccing for the moment, it will look like:

@mock.patch("car.Car")
def test_drive_2(self, mock_car):
    mock_car.return_value.drive.return_value = "BBBBBBBRRRRRR"
    noise = create_and_drive()
    assert(noise == "BBBBBBBRRRRRR")

This test passes when it shouldn’t. Add in the autospec argument like so:

@mock.patch("car.Car", autospec=True)

The test then correctly fails with the same error message:

TypeError: missing a required argument: 'speed'

As you can see if you autospec a class, then all the methods of that class are also specced.

One nuance to be aware of when speccing using a Class is that any attribute that is created after an object has been created will not be picked by autospec. For example, if I were to add an instance attribute to the car class like so:

class Car(object):
    def __init__(self, make):
        self.make = make
    def drive(self, speed):
        return "GOGOGOGO - {0}".format(speed)

Then change the create_and_drive function to just print out make:

def create_and_drive():
    new_car = Car("Porsche")
    print(new_car.make)
    return new_car.drive("FAST")

Then the following error occurs:

AttributeError: Mock object has no attribute 'make'

To get around this you change the test to assign that attribute to the mock object directly:

@mock.patch("car.Car", autospec=True)
def test_drive_2(self, mock_car):
    mock_car.return_value.drive.return_value = "BBBBBBBRRRRRR"
    mock_car.return_value.make = "Porsche"
    noise = create_and_drive()
    assert(noise == "BBBBBBBRRRRRR")

In conclusion, I recommend speccing your mock objects when writing tests but with the understanding of what it means to do so, as it reduces the risk of a test passing if the code is actually broken.

If you liked this article please recommend it, and I would love to hear any responses you have.