Python monkey-patching like a boss

If you work in a big project, most likely you will meet situations, when you would like to change/improve used third-party library behaviour, and you try to modify it from your project. This is called monkey-patching and it is usually associated with something not obvious, which would like to avoid. Nevertheless it’s a part of development process. And let’s see how to use monkey-patch valuably in Python.

Imagine that there is a third-party library my_lib with function func inside which behaviour I would like to change. And let’s go!

Note! Code below was verified with Python v3.7.

Patching module entities

To patch module (or class) variables or functions is easy, because Python doesn’t create protected or private entities and in most cases it’s possible to get reference to them.

1. Simple case — to replace result of function, patching variable value.

Third-party library structure:

my_lib/
└── __init__.py

__init__.py:

VAR = 'hello'
def func():
    return VAR + ' folks'

Let’s patch:

In [1]: import my_lib
In [2]: my_lib.VAR = 'bye-bye'
In [3]: my_lib.func()
Out[3]: 'bye-bye folks'

2. Patch value, when entities are in separated modules.

Library structure:

my_lib/
├── __init__.py
├── constants.py
└── functions.py

__init__.py:

# empty

constants.py:

VAR = 'hello'

functions.py:

from .constants import VAR
def func():
    return VAR + ' folks'

Let’s patch:

In [1]: from my_lib import constants
In [2]: constants.VAR = 'bye-bye'
In [3]: from my_lib import functions
In [4]: functions.func()
Out[4]: 'bye-bye folks'

3. Patch variable, when function is imported to package level from internal module.

The same sample as above, except one small thing, which changes all.

__init__.py:

from .functions import func

And patching doesn’t work then:

In [1]: from my_lib import constants
In [2]: constants.VAR = 'bye-bye'
In [3]: from my_lib import functions
In [4]: functions.func()
Out[4]: 'hello folks'

Oops…

Let’s look at the details.

Before package module import happens, Python always loads __init__.py of package. It means before from my_lib import constants, it imports my_lib/__init__.py where from .functions import func happens and as functions.py was imported, inside it also already constants.py was imported too. And from my_lib import constants doesn’t load the module, because it was loaded already, and func already refers to VAR = 'hello'.

In order to avoid such case, after VAR monkey-patching it needs to forcibly reload all affected packages and modules. In order to make it, it needs to delete them from cache of loaded modules with next function:

Let’s patch:

In [1]: from my_lib import constants
In [2]: constants.VAR = 'bye-bye'
In [3]: uncache(['my_lib.constants'])
In [4]: from my_lib import functions
In [5]: functions.func()
Out[5]: 'bye-bye folks'

And now it works also for package level too:

In [6]: import my_lib
In [7]: my_lib.func()
Out[7]: 'bye-bye folks'

For most cases it’s enough and can be used also to monkey-patch functions, classes, methods with the same approach. Main idea is to move reference from original to patched entity before its import or forcibly to reimport entity after patching.

But let’s play further!

Patching closed entities

Patching of closed entities inside functions is a bit tricky but also possible if to use ast and inspect modules.

Library structure:

my_lib/
└── __init__.py

1. Patch closed variable.

__init__.py:

def func():
    def _():
        var = 'hello'
        return var + ' folks' 
    return _()

Let’s patch:

In [1]: import my_lib
In [2]: import ast
In [3]: import inspect
In [4]: s = inspect.getsource(my_lib.func)
In [5]: m = ast.parse(s)
In [6]: m.body[0].body[0].body[0].value.s = 'bye-bye'
In [7]: co = compile(m, '<string>', 'exec')
In [8]: exec(co, my_lib.__dict__)
In [9]: my_lib.func()
Out[9]: 'bye-bye folks'

It’s more complicated than above variants, isn’t? Let’s learn it step-by-step:

• s = inspect.getsource(my_lib.func)— get source of patched function, because ast.parse doesn’t work with objects.
• m = ast.parse(s) — transform string code to Abstract Syntax Tree.
• m.body[0].body[0].body[0].value.s = ‘bye-bye’ — replace variable value in the tree.
• co = compile(m, '<string>', 'exec') — get Python code object.
• exec(co, my_lib.__dict__)— replace original function with patched.

2. Patch closed function.

__init__.py:

def func():
    def _():
        return 'hello folks'
    return _()

Let’s patch:

In [1]: import my_lib
In [2]: import ast
In [3]: import inspect
In [4]: m = ast.parse(inspect.getsource(my_lib.func))
In [5]: def _():
   ...:     return 'bye-bye folks'
In [6]: n = ast.parse(inspect.getsource(_))
In [7]: del _
In [8]: m.body[0].body[0] = n.body[0]
In [9]: exec(compile(m, '<string>', 'exec'), my_lib.__dict__)
In [10]: my_lib.func()
Out[10]: 'bye-bye folks'

3. Patch closed function inside class method.

__init__.py:

class My:
    def func(self):
        def _():
            return 'hello folks'
        return _()

Here direct ast.parse won’t work with inspect.getsource because method has additional indentation, which can’t be parsed:

In [1]: inspect.getsource(my_lib.My.func)
Out[1]: "    def func(self):\n        def _():\n            return 'hello folks'\n        return _()\n"
In [2]: ast.parse(inspect.getsource(my_lib.My.func))
Traceback (most recent call last):
...
File "<unknown>", line 1
  def func(self):
  ^
IndentationError: unexpected indent

Let’s patch with indentation cleaning:

In [1]: import my_lib
In [2]: import inspect
In [3]: import ast
In [4]: def source(o):
    ...:     s = inspect.getsource(o).split('\n')
    ...:     indent = len(s[0]) - len(s[0].lstrip())
    ...:     return '\n'.join(i[indent:] for i in s)
In [5]: m = ast.parse(source(my_lib.My.func))
In [6]: def _():
   ...:     return 'bye-bye folks'
In [7]: n = ast.parse(source(_))
In [8]: del _
In [9]: m.body[0].body[0] = n.body[0]
In [10]: exec(compile(m, '<string>', 'exec'))
In [11]: my_lib.My.func = func
In [12]: del func
In [13]: my_lib.My().func()
Out[13]: 'bye-bye folks'

Please pay attention in example above, exec(compile(m, ‘<string>’, ‘exec’), my_lib.My.__dict__) won’t work due to exception TypeError: exec() globals must be a dict, not mappingproxy. That’s why I was needed to exec in global namespace and then manually assign func as class method. Maybe there is another elegant variant, but I didn’t found.

Memory address changing

In internet also it’s possible to find semi-working samples how to change value of object by memory address management with module ctypes. So it’s highly not recommended to do, because it’s easy to break process with segmentation fault for example. Direct object memory manipulation is dangerous and not reliable, it can work for one value and will be broken for another.