How mock.patch decorator works in python

Published in

ukinau

5 min readFeb 23, 2018

I faced the trouble around mock.patch and parameterized and checked out how these works in general so I would write up here. This post will explain how mock.patch decorate in general and parameterized will be explained in other post later.

Cap: Usage of mock.patch

First of all let me cap the basic thing for mock.patch by writing simple python test.

  1 import os
  2 import mock
  3
  4
  5 @mock.patch("os.listdir", mock.MagicMock(return_value="test1"))
  6 def test1():
  7     assert "test1" == os.listdir()
  8
  9
 10 @mock.patch("os.listdir")
 11 def test2(mock_listdir):
 12     mock_listdir.return_value = "test2"
 13     assert "test2" == os.listdir()
 14
 15
 16 @mock.patch("os.listdir")
 17 class Test():
 18
 19     def not_decorated_and_not_tested(self):
 20         assert False
 21
 22     def test3(self, mock_listdir):
 23         mock_listdir.return_value = "test3"
 24         assert "test3" == os.listdir()

Basically we can replace the object which is specified as a first argument for “mock.patch” with MagicMock instance or any object you specified as a second argument. This is very useful when you write unit test to control other module behavior the module tested depends on.

If you executed above sample code, the output looks followings

~$ pytest test.py
=========== test session starts ===========
platform darwin — Python 2.7.12, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: /Users/test, inifile:
collected 3 itemstest.py …

“mock.patch” seems works correctly as you can see.

Actually there are many other way how to use patch. E.g. with “with statement” and with “start method, end method”… But this post focus on explaining decorator usage. Let’s dive into mock.patch decorator

The function to generate mock.patch decorator

Quoted important point from actual source code and bellow code is the actual decorator used when we write “@mock.patch(target)”

1598 def patch(
1599         target, new=DEFAULT, spec=None, create=False,
1600         spec_set=None, autospec=None, new_callable=None, **kwargs
1601     ):
1602     getter, attribute = _get_target(target)
1603     return _patch(
1604         getter, attribute, new, spec, create,
1605         spec_set, autospec, new_callable, kwargs
1606     )

Basically this function will generate the decorator function with “getter” which is the function to return actual object having attribute you wanted to replace and “attribute” which is the name of the object you want to replace. and then generate “_patch class” object which will be decorator with these.

Since probably explaining by only text is not easy thing so let me write python code do same thing as first example. bellow code is extracts from above sample code.

import os
import mock@mock.patch("os.listdir", mock.MagicMock(return_value="test1"))
def test1():
   assert "test1" == os.listdir()

This can be wrote as following

import os
import mockmockObj = mock.MagicMock(return_value="test1")
getter = lambda: __import__("os")
attribute = "listdir"decorator = mock.mock._patch(getter, attribute, mockObj, None, False, None, None, None, None)@decorator
def test1():
   assert "test1" == os.listdir()

If we disassemble more, the code will be

import os
import mockmockObj = mock.MagicMock(return_value="test1")
getter = lambda: __import__("os")
attribute = "listdir"decorator = mock.mock._patch(getter, attribute, mockObj, None, False, None, None, None, None)def test1():
   assert "test1" == os.listdir()test1 = decorator(test1)

Now it got more clear what mock.patch did, didn’t it?
But still we have big black box called as “mock.mock._patch” and This was the most interesting point, so move forwad to look at inside “mock.mock._patch”.

Inside decorator (mock.mock._patch)

This seems the function by assuming from the name that the name of this callable object doesn’t start from capital character, but actually this is the class to implement patch core logic. Although I said there are many way to use/call “mock.patch” , their differences are just how to call(frontend part) and all methods eventually use “mock.mock._patch” class to achieve patch logic, So reading this is meaningful if you are interesting to patch internal.

Let’s see the code, but one thing I have to remind you. I edited the code a lot and even more the code be shown from here can not be executed so think it as python like pseudo code. I wanted to make it simple as much as possible and make it easy understandable by this pseudo code. If you want to read actual code, go to github.

python/cpython

cpython - The Python programming language

github.com

class _patch(object):
    # The object of this class is decorator used like @_patch_object    attribute_name = None
    _active_patches = []    def __init__(
            self, getter, attribute, new, spec, create,
            spec_set, autospec, new_callable, kwargs
        ):
        # set all argument to instance variables    def __call__(self, func):
        # the decorator support class and function
        if isinstance(func, ClassTypes):
            return self.decorate_class(func)
        return self.decorate_callable(func)    def decorate_class(self, klass):
        for attr in dir(klass):
            if not attr.startswith(patch.TEST_PREFIX):
                # decorate only method having name start test_
                continue             attr_value = getattr(klass, attr)             patcher = self.copy()             # decorate function with same decorator
            setattr(klass, attr, patcher(attr_value))
        return klass     def decorate_callable(self, func):
        # If function is already decorated by patched function by
        # other patch decorator, this will just add patch obj 
        # pathings list without return new function
        if hasattr(func, 'patchings'):
            func.patchings.append(self)
            return func        @wraps(func)
        def patched(*args, **keywargs):
            extra_args = []
            entered_patchers = []
            try:
                for patching in patched.patchings:
                    arg = patching.__enter__() # replace
                    entered_patchers.append(patching) # For rollback
                    extra_args.append(arg) # Pass mock to decorated method
                args += tuple(extra_args)
                return func(*args, **keywargs)
      
            finally:
               # Rollback
               for patching in reversed(entered_patchers):
                    patching.__exit__(*exc_info)         # Initialize patchings attribute to store patch object
         patched.patchings = [self]
         return patched

I added a lot of comments so I want you to read these before move forward. As a complementary let me list up the points I want for you to notice.

This decorator (instance of “class _patch”) support class and function
If decorator used for class, the method having name starting “test_” of that class will be decorated by copied decorator (internally just use function decorator)
The function decorator (def decorate_callable)’s behavior would change depending on how many patch you are using for a method
First patch decoration will return new internal method(def patched) which have “patchings” list including self, but from second patch decoration, this patch decoration will just add self to “patchings” list of decorated function and return original function.
patched function which is returned when patch decoration evaluated first time basically check “patchings” list and try to replace all objects in that list, this iteration logic for “patchings” list enable the user to specify multiple patch decorator.

This is the overview of how mock.patch applied and if you understand up to here, probably it’s not difficult to know more detail like how to replace and rollback…and whatever…Just read the code if you want.

As a last thing, I would write the python script that explain what I explained

import os
import mock
mockObj = mock.MagicMock(return_value="test1")
getter = lambda: __import__("os")
attribute = "listdir"
decorator = mock.mock._patch(getter, attribute, mockObj, None, False, None, None, None, None)def test1():
   assert "test1" == os.listdir()print "Before", id(test1)
test1=decorator(test1)print "First decorate", id(test1)
test1=decorator(test1)print "Second decorate", id(test1)
test1=decorator(test1)print "test1.patchings", test1.patchings

This is created based on previous sample. This script is for checking that the function is changed or not by “id” builtin function and check “patchings” list of function as a last step. So let’s see the result

$ python test.py
Before 4506785744
First decorate 4514980040
Second decorate 4514980040
test1.patchings [<mock.mock._patch object at 0x10d172710>, <mock.mock._patch object at 0x10d172710>, <mock.mock._patch object at 0x10d172710>]

As you can see, the id of test1 was 4506785744 before apply any patch and then changed to 4514980040 after patch applied first time but second patch didn’t change id of the function, which proved function has not been changed from the time to apply patch first time. The last step proved test1 function have all patch object I applied in 3 steps.

Summary

I have used mock and patch for an age when write unit test but I have not read through the code before, So reading it was very interesting and I found out some information I didn’t know before. more specifically how patch decorate/change my original function is good insight when I write code.

Next post will be again the explanation for one of the test related python libraries “parameterized”