Python “parameterized” module is useful but some limitations are there

—
Updates(12 March 2018):
My patch to support mock.patch class decorator has been merged so now it support. https://github.com/wolever/parameterized/pull/53
but I left article as it was as the resource to explain internals of parameterised
—

When you want to run a certain test with multiple parameters, you will prepare the tests for each parameter if you think it in a straight forward way. This make readability worse and loose the maintainability.

This can be solved by generating the function bound with parameter from original function using decorator pattern and This is already developed by the someone pioneer.

wolever/parameterized

Basically all you have to do is to just write as following

from parameterized import parameterized
@parameterized.expand([[1,2], [3,4]])
def test1(test1, test2):
   print test1, test2
   assert False

In order to show standard output(print statement) I intentionally inserted “assert False”, let’s the output

test.py FF                                                                  [100%]
================ FAILURES =================
_____________ test1_1 _________________
@wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)
.pyenv/versions/2.7.12/lib/python2.7/site-packages/parameterized/parameterized.py:392: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test1 = 3, test2 = 4
@parameterized.expand([[1,2], [3,4]])
    def test1(test1, test2):
       print test1, test2
>      assert False
E      assert False
test.py:6: AssertionError
------------ Captured stdout call ---------------
3 4
_________ test1_0 __________________________
a = ()
@wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)
.pyenv/versions/2.7.12/lib/python2.7/site-packages/parameterized/parameterized.py:392: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
test1 = 1, test2 = 2
@parameterized.expand([[1,2], [3,4]])
    def test1(test1, test2):
       print test1, test2
>      assert False
E      assert False
test.py:6: AssertionError
-------------- Captured stdout call -------------
1 2
============= 2 failed in 0.10 seconds =====================

We can confirm that test1 function has not been executed and test1_0 and test1_1 functions have been generated with specified parameter and executed. It’s very useful but as you know it, this has been achieved by decorator and parameterized.expand decorator has mangled our original function and bring some limitations the developer definitely have to know.

From here, the main part of this article :)

Limitation of parameterized.expand (As of 24 February 2018)

Actually parameterized decorator (not parameterized.expand) also can be used but the available options are different between these and parameterized.expand decorator is useful than that, So this post only feature paramterized.expand

If you are in rushing to know, just check following summary

1. “parameterized.expand” convert original method to None object
2. “parameterized.expand” decorator should be defined outermost decorator for the target method if you use it with other decorator like (mock..)
3. When you use “parameterized.expand” with other class decorator which manipulate the methods of the class and that mapipulation logic refer/manipulate to any attributes of function like “patchings” in mock.patch , class decorator could work unexpectedly. Speaking more specifically, the multiple functions generated by “parameterized.expand” has same reference to attributes of original functions (wraps), but generated functions is different each other from class decorator point of view, thus class decorator would try to evaluate these methods respectively which have same reference to the attributes of original functions.

Probably all limitations look understandable to you except for last limitation. The last one looks rare/uncommon to you, but there is a combination we hit this limitation probably you will face. That is “parameterized.expand” and “patch” class decorator. We will deep dive what is happened exactly and how to avoid the issue later. Let’s see each limitation details first.

Why we can’t put it any middle of decorator chains

“parameterized.expand” decorator return None object(first limitation) thus we can not write any other decorator outer from “parameterized.expand” because outer decorator try to decorate the output from just former decorator, which means the outer decorator from “parameterized.expand” decorator will decorate None object “parameterized.expand” decorator return.

This explanation would link to 1, 2 limitations.

“parameterized.expand” and “mock.patch” doesn’t work…

This explanation would link to third limitation. First of all please look at following three code

The patch class decorator. It works well.

import os
from mock import patch
from parameterized import parameterized
@patch("os.listdir")
class Test():
    def test1(self, mock_listdir):
        pass

The patch class decorator with function decorator. It works well.

import os
from mock import patch
from parameterized import parameterized
@patch("os.listdir")
class Test():
    @patch("sys.path")
    def test1(self, mock_syspath, mock_listdir):
        pass

The patch and parameterized function decorator. It works well.

NB: we have to put patch decorator before parameterized due to second limitation

import os
from mock import patch
from parameterized import parameterized
class Test():
    @parameterized.expand([[1,2], [3,4], [5,6]])
    @patch("sys.path")
    def test1(self, t1, t2, mock_syspath):
        pass

Up to here, all three code work well. So How about following code?
Does it also work well?

import os
from mock import patch
from parameterized import parameterized
@patch("os.listdir")
class Test():
    @parameterized.expand([[1,2], [3,4], [5,6]])
    @patch("sys.path")
    def test1(self, t1, t2, mock_syspath, mock_listdir):
        pass

Unfortunately it doesn’t work well…What kind of error we got?

    @wraps(func)
    def patched(*args, **keywargs):
        extra_args = []
        entered_patchers = []
        exc_info = tuple()
        try:
            for patching in patched.patchings:
                arg = patching.__enter__()
                entered_patchers.append(patching)
                if patching.attribute_name is not None:
                    keywargs.update(arg)
                elif patching.new is DEFAULT:
                    extra_args.append(arg)
            args += tuple(extra_args)
>           return func(*args, **keywargs)
E           TypeError: test1() takes exactly 5 arguments (7 given)
.pyenv/versions/2.7.12/lib/python2.7/site-packages/mock/mock.py:1305: TypeError

Above error is the interesting extract from the whole error logs. This error claim the number of arguments are not same as definition of method.
So let us look at what is additional argument by adding argument for debug

New code will be

import os
from mock import patch
from parameterized import parameterized
@patch("os.listdir")
class Test():
    @parameterized.expand([[1,2], [3,4], [5,6]])
    @patch("os.mkdir")
    def test1(self, t1, t2, mock_mkdir, what1, what2, what3):
        import pdb; pdb.set_trace()
        pass

Execute this code and start pdb debugger

>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>
> /Users/ukinau/test.py(13)test1()
-> pass
(Pdb) l
  8
  9         @parameterized.expand([[1,2], [3,4], [5,6]])
 10         @patch("os.mkdir")
 11         def test1(self, t1, t2, mock_mkdir, what1, what2, what3):
 12             import pdb; pdb.set_trace()
 13  ->         pass
 14
 15
(Pdb) what1
<MagicMock name='listdir' id='4457883856'>
(Pdb) what2
<MagicMock name='listdir' id='4457917520'>
(Pdb) what3
<MagicMock name='listdir' id='4457938896'>

Oh, 3 different “listdir” mock arguments are passed to this method…we expected just one though…
In order to grasp why this happened, you’d better to read How mock.patch decorator works in python first and come back here again.

I described how decorator change test1 method of Test class to understand problem

Let me explain each in 3 steps

1. function patch decorator

• Create .patchings list and return new function to use .patching list if original function doesn’t have it
• Add patch object to .patching list

2. function expand decorator

• Generate new function bound with argument for each parameter
• New function will be registered to class by manipulating local namespace directly by inspect
• Copy __dict__ attribute of original function to new function by wraps decorator. This will copy just value so all generated function have same reference of .patchings list original function had
• Change original function from function object to None object by returning None from expand decorator

If you want to more explicitly, look at following line, this link bring you right place.

wolever/parameterized

3. class patch decorator

• All callable functions having name starting from “test_” are decorated with function patch decorator
• function patch decorator add patch object to original function’s pathings list

The problem is that “expand function decorator” generate multiple functions having same reference to patchings list and “mock.patch” evaluated these as the method having reference to different patchings list.

After all decorator evaluated, the state of Class will be like following

This is the why the mock objects which are injected by patch class decorator are passed to function in multiple times than we expected.

If you understand up to here, you can easily avoid this problem by changing parameterized library, I resolved this problem as following.

Support mock.patch class decorator · ukinau/parameterized@3bcc172

If you applied my patch or use my branch of parameterized, you can run following code without any error.

import os
from mock import patch
from parameterized import parameterized
@patch("os.uname")
@patch("os.listdir")
class Test():
    @parameterized.expand([[1,2], [3,4], [5,6], [7,8]])
    @patch("os.mkdir")
    @patch("os.umask")
    def test1(self, t1, t2, mock_umask, mock_mkdir, mock_listdir, mock_uname):
        mock_umask.return_value = 1
        mock_mkdir.return_value = 2
        mock_listdir.return_value = 3
        mock_uname.return_value = 4
        assert  t1 + 1 == t2
        assert 1 == os.umask()
        assert 2 == os.mkdir()
        assert 3 == os.listdir()
        assert 4 == os.uname()

Really works? yes worked.

pytest test.py
====== test session starts ===========
platform darwin -- Python 2.7.12, pytest-3.4.0, py-1.5.2, pluggy-0.6.0
rootdir: /Users/ukinau, inifile:
collected 4 items
test.py ....                                                                                                                                                                  [100%]
======== 4 passed in 0.14 seconds =======

Summary

In this post I wrote the limitation/specification of parameterized module and I tried to resolve one of the limitations which looks bug to me. Actually I was going to send PR for that but this can be thought as too specific usecase (mock.patch is the cpython standard library nowadays though), this thought make me hesitate but I will try it although I don’t have enough confidence for this patch to be merged.