The Nine Circles of Python Dependency Hell

Knewton
Knewton
Sep 28, 2015 · 7 min read

Written by Paul Kernfeld

“Dependency hell” is a term for the frustration that arises from problems with transitive (indirect) dependencies. Dependency hell in Python often happens because pip does not have a dependency resolver and because all dependencies are shared across a project.1 In this post, I’ll share a few of the strategies that I use to deal with some commonly-encountered problems.

These strategies assume that you’re using a dependency management setup similar to what we use at Knewton, which includes using pip, virtualenv, and good practices for your requirements.txt and install_requires. Some items are specific to organizations that use both internal and external Python libraries, but many of these items will apply to any Python project.

Detecting dependency hell

Even if your project only has a few first-level dependencies, it could have many more transitive dependencies. This means that figuring out that you even have a version conflict is non-trivial. If you just run pip install -r requirements.txt, pip will happily ignore any conflicting versions of libraries. In the best case, your project will work fine and you won’t even notice that this has happened. If you’re unlucky, though, you’ll get a mysterious runtime error because the wrong version of a library is installed.

In this case, my_app depends on foo_client and bar_client. Let’s assume that bar_client uses a feature that was introduced after requests 2.3.1. If pip installs requests==2.3.1 (from foo_client), bar_client will break because the feature it needs is missing! Note that foo_client and bar_client can each build fine independently.
In this case, my_app depends on foo_client and bar_client. Let’s assume that bar_client uses a feature that was introduced after requests 2.3.1. If pip installs requests==2.3.1 (from foo_client), bar_client will break because the feature it needs is missing! Note that foo_client and bar_client can each build fine independently.

In this case, my_app depends on foo_client and bar_client. Let’s assume that bar_client uses a feature that was introduced after requests 2.3.1. If pip installs requests==2.3.1 (from foo_client), bar_client will break because the feature it needs is missing!
Note that foo_client and bar_client can each build fine independently.

Stable strategy: pip-conflict-checker and pipdeptree

See python-project-template-with-pip-conflict-checker on Github for an explanatory template project that uses this strategy.

If you need a more detailed view of which dependencies are conflicting with each other, try installing and running pipdeptree. It will produce an output like this, helpfully highlighting any possible conflicts:

$ pipdeptree
Warning!!! Possible confusing dependencies found:
* Mako==0.9.1 -> MarkupSafe [required: >=0.9.2, installed: 0.18]
Jinja2==2.7.2 -> MarkupSafe [installed: 0.18]
------------------------------------------------------------------------
Lookupy==0.1
wsgiref==0.1.2
argparse==1.2.1
psycopg2==2.5.2
Flask-Script==0.6.6
- Flask [installed: 0.10.1]
- Werkzeug [required: >=0.7, installed: 0.9.4]
- Jinja2 [required: >=2.4, installed: 2.7.2]
- MarkupSafe [installed: 0.18]
- itsdangerous [required: >=0.21, installed: 0.23]
alembic==0.6.2
- SQLAlchemy [required: >=0.7.3, installed: 0.9.1]
- Mako [installed: 0.9.1]
- MarkupSafe [required: >=0.9.2, installed: 0.18]
ipython==2.0.0
slugify==0.0.1
redis==2.9.1

[output courtesy of the pipdeptree page]

Experimental strategy: pip-compile

The only disadvantage of pip-compile is that it’s still immature. At the time of writing, it’s lacking some important features, like support for extras.

See python-project-template-with-pip-compile for an explanatory template project that uses this strategy.

Fixing Dependency Conflicts

Circle 1: Unnecessary dependencies

Circle 2: Old dependencies

Circle 3: Transitive dependencies conflict with requirements.txt

Circle 4: Overlapping transitive dependencies

To fix this problem, use a constraints file to specify the version range that will satisfy both dependencies.2,3 Requirements.txt and constraints files allow comments using the # character, so you can, and should, document why you’re constraining the versions of each package.

If you’re using pip-compile, it will automatically do version resolution for you, so you don’t need to worry about this.

In this case, my_app depends on fussy_foo and capricious_client, which depend on overlapping versions of requests. By looking at the dependency diagram, we can see that requests>=1.2.0,</a> In this case, my_app depends on fussy_foo and capricious_client, which depend on overlapping versions of requests. By looking at the dependency diagram, we can see that requests>=1.2.0,<2 will satisfy both clients. If capricious_client’s version of requests is installed, its version of requests will be used, which will probably be fine, since pip will pick the most recent version of requests satisfying the constraints. If fussy_foo’s version of requests is installed, though, it’ll install a version of requests that will be too new for capricious_client.[/caption]</p><p>[caption id=
In this case, my_app depends on fussy_foo and capricious_client, which depend on overlapping versions of requests. By looking at the dependency diagram, we can see that requests>=1.2.0,</a> In this case, my_app depends on fussy_foo and capricious_client, which depend on overlapping versions of requests. By looking at the dependency diagram, we can see that requests>=1.2.0,<2 will satisfy both clients. If capricious_client’s version of requests is installed, its version of requests will be used, which will probably be fine, since pip will pick the most recent version of requests satisfying the constraints. If fussy_foo’s version of requests is installed, though, it’ll install a version of requests that will be too new for capricious_client.[/caption]</p><p>[caption id=
Adding a top-level constraint on the version of requests can satisfy all dependencies.
Adding a top-level constraint on the version of requests can satisfy all dependencies.

Adding a top-level constraint on the version of requests can satisfy all dependencies.

Circle 5: Internal transitive dependency conflict

Circle 6: External dependency conflict in name only

This may be a “non-problem” in that the projects are probably actually compatible. If you’re using pip-conflict-checker, you could just turn it off in your build. Similarly, although pip-compile will complain, you could just manually write out your requirements.txt file. These solutions are hacky and should make you a little uncomfortable. If you want to do this “the right way,” you’ll probably need to fork one of the projects and modify it to use a less restrictive version range.

In this case, picky_project and pinning_project have each pinned requests to an overly-specific version in their install_requires.
In this case, picky_project and pinning_project have each pinned requests to an overly-specific version in their install_requires.

In this case, picky_project and pinning_project have each pinned requests to an overly-specific version in their install_requires.

Circle 7: Monolithic project

In this example, my_project consists of a separate server and client. The server depends on flask, and the client depends on matplotlib. Each of these packages brings in its own dependencies, which creates a large tree.
In this example, my_project consists of a separate server and client. The server depends on flask, and the client depends on matplotlib. Each of these packages brings in its own dependencies, which creates a large tree.

In this example, my_project consists of a separate server and client. The server depends on flask, and the client depends on matplotlib. Each of these packages brings in its own dependencies, which creates a large tree.

After splitting the project into separate server and client portions, we end up with two small projects instead of one big project. All right!
After splitting the project into separate server and client portions, we end up with two small projects instead of one big project. All right!

After splitting the project into separate server and client portions, we end up with two small projects instead of one big project. All right!

Circle 8: Seriously incompatible dependencies

Circle 9: Something else entirely

Python build tools are very modular, so you may need to learn separately about wheels, older binary distribution formats, PyPI, or whatever else is part of your toolchain. Good luck!

Notes

  1. Some dependency management systems, like npm, avoid this problem by making all dependencies dependency-local instead of project-local. Others create renamed versions of dependencies (e.g., “shading” in Maven) to work around dependency hell.
  2. You might be able to fix this problem by reordering your requirements.txt file, since pip does resolve requirements in order. This is not a good idea because pip does not guarantee that requirements will be resolved in a specific order; it’s just a byproduct of pip’s current implementation.
  3. Constraints files are new in pip 7.1, so if you need to use an older version of pip, you can accomplish the same thing by adding the correct version of the transitive dependency to your requirements.txt file. This works because pip prioritizes requirement versions in breadth-first order.

Knerd

The Knewton Blog - Stories about technology, product and…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store