The Nine Circles of Python Dependency Hell

Written by Paul Kernfeld

“Dependency hell” is a term for the frustration that arises from problems with transitive (indirect) dependencies. Dependency hell in Python often happens because pip does not have a dependency resolver and because all dependencies are shared across a project.1 In this post, I’ll share a few of the strategies that I use to deal with some commonly-encountered problems.

These strategies assume that you’re using a dependency management setup similar to what we use at Knewton, which includes using pip, virtualenv, and good practices for your requirements.txt and install_requires. Some items are specific to organizations that use both internal and external Python libraries, but many of these items will apply to any Python project.

Detecting dependency hell

Even if your project only has a few first-level dependencies, it could have many more transitive dependencies. This means that figuring out that you even have a version conflict is non-trivial. If you just run pip install -r requirements.txt, pip will happily ignore any conflicting versions of libraries. In the best case, your project will work fine and you won’t even notice that this has happened. If you’re unlucky, though, you’ll get a mysterious runtime error because the wrong version of a library is installed.

In this case, my_app depends on foo_client and bar_client. Let’s assume that bar_client uses a feature that was introduced after requests 2.3.1. If pip installs requests==2.3.1 (from foo_client), bar_client will break because the feature it needs is missing!
Note that foo_client and bar_client can each build fine independently.

Stable strategy: pip-conflict-checker and pipdeptree

Pip-conflict-checker is a great tool for quickly determining whether you have any version conflicts. It recursively checks the requirements of each library and prints out any conflicts. It can be used to fail your build when a dependency conflict is found.

See python-project-template-with-pip-conflict-checker on Github for an explanatory template project that uses this strategy.

If you need a more detailed view of which dependencies are conflicting with each other, try installing and running pipdeptree. It will produce an output like this, helpfully highlighting any possible conflicts:

$ pipdeptree
Warning!!! Possible confusing dependencies found:
* Mako==0.9.1 -> MarkupSafe [required: >=0.9.2, installed: 0.18]
Jinja2==2.7.2 -> MarkupSafe [installed: 0.18]
------------------------------------------------------------------------
Lookupy==0.1
wsgiref==0.1.2
argparse==1.2.1
psycopg2==2.5.2
Flask-Script==0.6.6
- Flask [installed: 0.10.1]
- Werkzeug [required: >=0.7, installed: 0.9.4]
- Jinja2 [required: >=2.4, installed: 2.7.2]
- MarkupSafe [installed: 0.18]
- itsdangerous [required: >=0.21, installed: 0.23]
alembic==0.6.2
- SQLAlchemy [required: >=0.7.3, installed: 0.9.1]
- Mako [installed: 0.9.1]
- MarkupSafe [required: >=0.9.2, installed: 0.18]
ipython==2.0.0
slugify==0.0.1
redis==2.9.1

[output courtesy of the pipdeptree page]

Experimental strategy: pip-compile

Pip-compile, from pip-tools, is a tool that takes in a file with requirements ranges and spits out a requirements.txt file with pinned versions. To do this correctly, it actually looks at transitive dependencies, does version resolution, and even reports version conflicts when they occur! This makes it trivial to keep your requirements.txt and install_requires in sync. In fact, several of the problems listed below can’t even happen if you’re using pip-compile!

The only disadvantage of pip-compile is that it’s still immature. At the time of writing, it’s lacking some important features, like support for extras.

See python-project-template-with-pip-compile for an explanatory template project that uses this strategy.

Fixing Dependency Conflicts

After you’ve found a dependency conflict, well … now what?

Circle 1: Unnecessary dependencies

It’s easy to forget to remove unused dependencies from your requirements.txt after removing the code that uses them. If your project has some old dependencies that are no longer used, eliminating them could solve your problem, but if it doesn’t, at the very least it should make your builds faster.

Circle 2: Old dependencies

Upgrading to newer versions of your dependencies may solve the problem. A single old dependency can bring in a whole web of old transitive dependencies.

Circle 3: Transitive dependencies conflict with requirements.txt

You may see a version conflict if your dependencies are demanding a specific version of a library that you don’t currently use in your own requirements.txt. The best solution for this is to pin your requirements.txt file to a version of the library that works for both your project and for your dependencies. If you’re using pip-compile, you can do this by modifying your requirements.in file and then recompiling your requirements.txt file.

Circle 4: Overlapping transitive dependencies

If two of your dependencies are demanding overlapping versions of a library, pip will not necessarily install a version of this library that satisfies both requirements! This shouldn’t happen when installing from a requirements.txt file, but it can occur when trying to update your requirements.txt file.

To fix this problem, use a constraints file to specify the version range that will satisfy both dependencies.2,3 Requirements.txt and constraints files allow comments using the # character, so you can, and should, document why you’re constraining the versions of each package.

If you’re using pip-compile, it will automatically do version resolution for you, so you don’t need to worry about this.

Adding a top-level constraint on the version of requests can satisfy all dependencies.

Circle 5: Internal transitive dependency conflict

You may run into a situation where two of your internal dependencies are demanding non-overlapping versions of a library. To fix this, modify one or both of your internal dependencies to be less specific in its install_requires. Remember that, unlike requirements.txt, install_requires should allow wide version bands.

Circle 6: External dependency conflict in name only

Occasionally, you may see a situation where two external dependencies have specified very slightly different versions of a transitive dependency in their install_requires.

This may be a “non-problem” in that the projects are probably actually compatible. If you’re using pip-conflict-checker, you could just turn it off in your build. Similarly, although pip-compile will complain, you could just manually write out your requirements.txt file. These solutions are hacky and should make you a little uncomfortable. If you want to do this “the right way,” you’ll probably need to fork one of the projects and modify it to use a less restrictive version range.

In this case, picky_project and pinning_project have each pinned requests to an overly-specific version in their install_requires.

Circle 7: Monolithic project

If your project has too many dependencies, maybe it shouldn’t be a single project! It’s much easier said than done, but breaking up a gigantic project can improve dependency management, install and test time, and ease of comprehension.

In this example, my_project consists of a separate server and client. The server depends on flask, and the client depends on matplotlib. Each of these packages brings in its own dependencies, which creates a large tree.

After splitting the project into separate server and client portions, we end up with two small projects instead of one big project. All right!

Circle 8: Seriously incompatible dependencies

In a particularly sad situation, you may find that you have two dependencies that depend on very incompatible versions of a common transitive dependency. Ideally, you should try to update one of your dependencies to use a newer version of the shared library. If you can’t do this, a good backup plan is to unleash a tormented wail and weep into your keyboard.

Circle 9: Something else entirely

You will inevitably run into installation problems that are not listed above (or on StackOverflow!). As with any similar situation, the only really good way to remedy this situation is to understand thoroughly the toolchain you’re using.

• The Python Packaging User Guide: an excellent overview of Python packaging.
• Pip’s User Guide: a good overview of pip, with nice explanation of version conflicts.
• The pip install documentation: a deeper dive into pip install.
• The tox configuration documentation: a bit sparse, but it can be a lifesaver.
• PEP 0440, the standard for package naming: watch out for this because newer versions of pip are picky about adhering to the naming standard.

Python build tools are very modular, so you may need to learn separately about wheels, older binary distribution formats, PyPI, or whatever else is part of your toolchain. Good luck!

Notes

1. Some dependency management systems, like npm, avoid this problem by making all dependencies dependency-local instead of project-local. Others create renamed versions of dependencies (e.g., “shading” in Maven) to work around dependency hell.
2. You might be able to fix this problem by reordering your requirements.txt file, since pip does resolve requirements in order. This is not a good idea because pip does not guarantee that requirements will be resolved in a specific order; it’s just a byproduct of pip’s current implementation.
3. Constraints files are new in pip 7.1, so if you need to use an older version of pip, you can accomplish the same thing by adding the correct version of the transitive dependency to your requirements.txt file. This works because pip prioritizes requirement versions in breadth-first order.