Pythonic Dependency Injection: A Practical Guide

Back in the 90s Bob Martin came up with a particularly simple yet useful principle for decoupling software components:

  • High-level modules should not depend on low-level modules. Both should depend on abstractions.
  • Abstractions should not depend on details. Details should depend on abstractions.

In the SOLID framework for software development, this came to be known as the dependency inversion principle. Dependency injection is a design pattern that supports designing software components that follow this principle.

In this article we’ll be looking into a few options for doing dependency injection in Python, here-among a useful framework for Python 3 with PEP484 support. As a running example we’ll be writing a program that remote controls a fence guarding robot through a web API.

For that, we ‘ll write a small API client that uses the requests library:

import requests
class RobotApi:
url = 'http://robot-api.com'

def send_command(self, command, data):
formatted_url = f'{self.url}/{command}'
requests.post(formatted_url, data=data)

For controlling the robot, we write a general purpose class for high level movement operations:

class RobotControls:
def __init__(self, api):
self.api = api
    def move_west(self):
self.api.send_command('move_west', data={'meters': 1})

def move_east(self):
self.api.send_command('move_east', data={'meters': 1})

def move_north(self):
self.api.send_command('move_north', data={'meters': 1})

def move_south(self):
self.api.send_command('move_south', data={'meters': 1})

In this example the RobotControls class implicitly depends on an abstract interface that defines a send_command method. Since RobotControls doesn’t use any imports but instead receives this dependency through its constructor, we rather grandiosely say that the dependency is injected.

Injecting dependencies in this way has a number of advantages:

  • The RobotControls class becomes configurable and thus more reusable. By changing the api instance injected through __init__ we can change the overall behavior of RobotControls
  • The RobotControls class becomes easier to unit-test, especially if api or any of api‘s dependencies uses IO operations because we can easily replace that dependency with a version that mocks or stubs the IO.
  • Using dependency injection encourages us to pay more attention to what responsibilities belong in RobotControls and what responsibilities belong in its dependencies thus increasing overall cohesion and encapsulation of the units in our program.

The main disadvantage of injecting dependencies through __init__ is that it requires us to construct an object that has a send_command method before we can instantiate a RobotControls. This may not seem like a huge problem in our example, but consider the case when we want to inject RobotControls in a new class, and we want to inject that class as a dependency in another class and so on and so on. Take for example our FenceGuardingRobot class:

class FenceGuardingRobot:
fence_length = 20
    def __init__(self, robot_controls):
self.robot_controls = robot_controls
    def guard(self):
for _ in range(self.fence_length):
self.robot_controls.move_north()
for _ in range(self.fence_length):
self.robot_controls.move_south()

In order to instantiate this class, we now have to write:

fence_guarding_robot = FenceGuardingRobot(RobotControls(RobotApi()))

The annoyance of having to construct such a dependency graph manually is one reason why good intentions of using dependency injection pervasively in many projects often die as the code base grows large. You can of course use default parameter values to reduce this problem, but then you run the risk of programming against concretions rather than abstractions.

One way of avoiding this pain is by using a pattern built around the super keyword in Python. The super keyword has a slightly different meaning in Python than in many other object-oriented languages. When calling super, Python will compute a so-called linearization of the base classes of the calling class, which simply means that Python figures out in which order it should look up names in the base classes. In Python this is referred to as method resolution order. We can change our example to use dependency injection through super as follows:

import requests


class RobotApi:
url = 'http://robot-api.com'

def send_command(self, command, data):
formatted_url = f'{self.url}/{command}'
requests.post(formatted_url, data=data)


class RobotControls(RobotApi):
def move_west(self):
super().send_command('move_west', data={'meters': 1})

def move_east(self):
super().send_command('move_east', data={'meters': 1})

def move_north(self):
super().send_command('move_north', data={'meters': 1})

def move_south(self):
super().send_command('move_south', data={'meters': 1})


class FenceGuardingRobot(RobotControls):
fence_length = 20

def guard(self):
for _ in range(self.fence_length):
super().move_north()
for _ in range(self.fence_length):
super().move_south()

You can see a type’s method resolution order using the builtin help method:

help(FenceGuardingRobot)

This outputs:

class FenceGuardingRobot(RobotControls)
| Method resolution order:
| FenceGuardingRobot
| RobotControls
| RobotApi
| builtins.object
|
| Methods defined here:
|
| guard(self)
|
| ----------------------------------------------------------------------
| Data and other attributes defined here:
|
| fence_length = 20
|
| ----------------------------------------------------------------------
| Methods inherited from RobotControls:
|
| move_east(self)
|
| move_north(self)
|
| move_south(self)
|
| move_west(self)
|
| ----------------------------------------------------------------------
| Methods inherited from RobotApi:
|
| send_command(self, command, data)
|
| ----------------------------------------------------------------------
| Data descriptors inherited from RobotApi:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
|
| ----------------------------------------------------------------------
| Data and other attributes inherited from RobotApi:
|
| url = 'http://robot-api.com'

In the Method resolution order section, we can see that when looking up names using super, Python will look for the name in FenceGuardingRobot first, then RobotControls, then RobotApi, then object.

Now, to instantiate FenceGuardRobot, all we have to do is write FenceGuardRobot(). If we want to replace a dependency for FenceGuardRobot, for example to mock out the RobotApi class, all we have to do is define a new type that adds a new send_command method earlier in the linearization:

from unittest.mock import Mock


class MockRobotApi(RobotApi):
send_command = Mock()


class MockedFenceGuardingRobot(FenceGuardingRobot, MockRobotApi):
pass

If we inspect the method resolution order for MockedFenceGuardingRobot, we’ll see that MockRobotApi comes before RobotApi:

help(MockedFenceGuardingRobot)

outputs:

class MockedFenceGuardingRobot(FenceGuardingRobot, MockRobotApi)
| Method resolution order:
| MockedFenceGuardingRobot
| FenceGuardingRobot
| RobotControls
| MockRobotApi
| RobotApi
| builtins.object
...

Note that MockRobotApi must inherit from RobotApi in order for the method resolution to be in this order. This is a consequence of the algorithm used by python to construct the linearization.

The main advantage of this pattern using super for dependency injection over the simple approach where dependencies are injected through __init__ is that we don’t have to construct the dependency graph manually. All the wiring is handled by the semantics of super.

There are a few disadvantages to this approach. First and foremost, it becomes fairly unclear which names are provided by which dependencies. In our example it’s obvious because all classes have exactly one dependency, but when you have more than one dependency per class, things can become pretty oblique. For example, if FenceGuardingRobot had more than one dependency it wouldn’t be possible to tell which part of the class was being mocked out by MockedFenceGuardingRobot just by looking at the definitions.

Secondly, all the classes participating in this pattern must be designed for multiple inheritance. Specifically, if a class takes arguments through __init__ it must take care to also take *args and **kwargs and call super().__init__(*args, **kwargs) even if that class doesn’t inherit from anything since your injected class may not be the last class in the linearization:

class A:
def __init__(self, a, *args, **kwargs):
self.a = a
super().__init__(*args, **kwargs)
class B:
def __init__(self, b):
self.b = b
super().__init__(*args, **kwargs)
class C(A, B):
pass

If you need to inject a class that is not designed for multiple inheritance in this way, you need to write an adapter class for it. See the original article for an example.

Lastly, you are no longer coding against abstractions but rather against concretions. You can use the abc module to write abstract classes in Python, but depending on which tool you use to check for unimplemented members, you may have to re-declare the abstract members of your dependencies as abstract in the dependent class.

As a third option, you can use a library. A number of possibilities exist, but I’ll present the one with which I‘m most familiar: serum (full disclosure: I’m the author).

serum attempts to take the best of both worlds from the previous two approaches to dependency injection: It takes care of all the wiring so you don’t have to manually construct the dependency graph yourself, but does so using composition rather than inheritance.

We can rewrite our example to use serum as follows:

from serum import inject, Component, Environment

class RobotApi(Component):
url = 'http://robot-api.com'

def send_command(self, command, data):
formatted_url = f'{self.url}/{command}'
requests.post(formatted_url, data=data)

class RobotControls(Component):
api = inject(RobotApi)
    def move_west(self):
self.api.send_command('move_west', data={'meters': 1})

def move_east(self):
self.api.send_command('move_east', data={'meters': 1})

def move_north(self):
self.api.send_command('move_north', data={'meters': 1})

def move_south(self):
self.api.send_command('move_south', data={'meters': 1})

class FenceGuardingRobot:
robot_controls = inject(RobotControls)
fence_length = 20
    def guard(self):
for _ in range(self.fence_length):
self.robot_controls.move_north()
for _ in range(self.fence_length):
self.robot_controls.move_south()

To mock out RobotApi we could do:

class MockRobotApi(RobotApi):
send_command = Mock()
with Environment(MockRobotApi):
assert isinstance(
FenceGuardingRobot().robot_controls.robot_api,
MockRobotApi
)

Or even

from serum import mock
from unittest.mock import MagicMock

with Environment():
mock(RobotApi)
assert isinstance(
FenceGuardingRobot().robot_controls.robot_api,
MagicMock
)

The main disadvantage of using serum rather than a language feature like super for dependency injection is that injectable classes must inherit from serum.Component which primarily adds the constraint that the __init__ method of the class can only take the self parameter. This is necessary to make the lazy dependency injection mechanism used by serum possible.

You can get around this by injecting dependencies by string names rather than by type, but this disables some PEP 484 related features of serum:

from serum import Environment, inject
class NotAComponent:
pass
instance = NotAComponent()
with Environment(dependency=instance):
assert inject('dependency') is instance

That concludes our tour of dependency injection mechanisms in Python. To summarize, you can use __init__ but that often leads to annoying code for wiring up your application. You can use super but that often leads to oblique dependencies in the sense that its hard to tell which dependencies provide which names. Finally you can use a library like serum that combines the best of both worlds. For more information, issue tracking and pull requests for serum, see the GitHub page.

Update 04/18/18

I just released version 4.0.0 of serum. This release aims to make the framework much less invasive in client code. In this version you can define your dependent classes and dependencies using annotation syntax that is pure python:

class RobotApi:
...
class FenceGuardingRobot:
api: RobotAPi
...

And then use decorators to start using serum:

from serum import dependency, inject
@dependency
class RobotApi:
...
@inject
class FenceGuardingRobot:
api: RobotApi

assert isinstance(FenceGuardingRobot().api, RobotApi)

And that’s it!