Photo by Elisa Ventur on Unsplash

How I fixed a pip-compile dependency resolution error

Just like you (right?) I lock my application’s Python dependencies, and I use pip-tools to do it. It’s a great tool that makes sure my full set of recursive dependencies is fully locked and that I’ll be able to replicate my install on all my systems. But I found that I needed to upgrade a particular package, and the standard procedure wasn’t working, and wasn’t giving a great error message. Here’s how I figured it out.

The error message

I am using Sentry (a great product, try it if you haven’t), and I badly need to be upgraded to the latest version of the sentry-sdk Python package. Unfortunately, I’m pretty far behind in a number of other dependencies, and I know that some of them will be pretty hairy to upgrade, so I don’t just want to upgrade everything in a big batch. I’ve got to take it a step at a time.

To that end, I ran pip-compile --upgrade-package sentry-sdk . And while I expected no other packages to be upgraded as pip-compile by default won’t change anything it doesn’t have to from an existing lock file, I found that it didn’t upgrade anything at all, including sentry-sdk. Something was amiss.

So I got more aggressive. I was on version 1.9.0 . I didn’t know what the latest version was, but I was okay with just upgrading a little bit at a time. Just give me something. So I ran pip-compile --upgrade-package 'sentry-sdk>1.9.0' . While I got a different result, it wasn’t quite the result I was hoping for. It was a rather hairy error.

Amid the noise, I found a couple of lines that seemed like they might be able to give me a hint about what was going on.

  ERROR: Cannot install -r requirements.in (line 44), -r requirements.in (line 80) and -r requirements.in (line 81) because these package versions have conflicting dependencies.
Discarding urllib3==1.25.11 (from -r requirements.txt (line 589)) to proceed the resolution
  ERROR: Cannot install -r requirements.in (line 44), -r requirements.in (line 80), -r requirements.in (line 81) and boto3 because these package versions have conflicting dependencies.

[... a lot of traceback ...]

pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Those are the remnants of some helpful pip error messages, but they’re not really helping me get to the bottom of what’s going on.

Enter pipdeptree

But I did notice a little something. It seemed like there was a conflict involving urllib3, and I know that lots of packages depend on that in some sort of way. So I ran pipdeptree a couple of ways, to help me get a better sense of what I might be looking at. pipdeptree works by looking at the currently installed packages, and letting you poke around to see what depends on what. I started by just confirming sentry-sdk's dependencies.

$ pipdeptree -p sentry-sdk
sentry-sdk==1.9.0
├── certifi [required: Any, installed: 2023.5.7]
└── urllib3 [required: >=1.10.0, installed: 1.25.11]

OK, not bad. I saw that I’m using sentry-sdk==1.9.0 and urllib3==1.25.11 . A quick check with pip told me that these are both out of date.

$ pip list --outdated | grep -e sentry-sdk -e urllib3
sentry-sdk               1.9.0       1.28.1   wheel
urllib3                  1.25.11     2.0.4    wheel

This made me really wonder what all I had installed that might be holding urllib3 back. So back I went to pipdeptree, but this time using it in reverse.

$ pipdeptree -r -d 1 -p urllib3
urllib3==1.25.11
├── botocore==1.14.17 [requires: urllib3>=1.20,<1.26]
├── google-auth==2.20.0 [requires: urllib3<2.0]
├── mailchimp-transactional==1.0.47 [requires: urllib3>=1.23]
├── requests==2.31.0 [requires: urllib3>=1.21.1,<3]
├── selenium==3.141.0 [requires: urllib3]
└── sentry-sdk==1.9.0 [requires: urllib3>=1.10.0]

What stood out to me here is that there are a couple of packages that are not permitting urllib3 to reach version 2.0. Finally I had a theory about what’s going wrong. I suspected that newer versions of sentry-sdk require a newer version of urllib3 than is installed, but some of my other packages require versions older than what sentry-sdk requires. No way to square that circle.

pip to the rescue

All that was just a hypothesis, though. In order to see if that was the case, I needed to get a newer version of sentry-sdk installed, so that pipdeptree could tell me its dependencies. So that’s what I did, and pip turned out to be more informative than I had expected.

$ pip install sentry-sdk --upgrade
Requirement already satisfied: sentry-sdk in /home/vscode/.local/lib/python3.10/site-packages (1.9.0)
Collecting sentry-sdk

[... a bunch of normal pip stuff ...]

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
botocore 1.14.17 requires urllib3<1.26,>=1.20; python_version != "3.4", but you have urllib3 2.0.4 which is incompatible.
google-auth 2.20.0 requires urllib3<2.0, but you have urllib3 2.0.4 which is incompatible.
Successfully installed sentry-sdk-1.28.1 urllib3-2.0.4

Sweet. I don’t even have to go back to pipdeptree to figure out where the version conflicts are. Pip just went ahead and installed an incompatible set of dependencies, but it gave me an awesomely helpful warning that there were three packages that were not compatible with the new versions I’d just installed. Those packages were botocore and google-auth. Well let’s just see what happens if we upgrade those!

$ pip install --upgrade botocore google-auth

[... same old same old ...]

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.11.9 requires botocore<1.15.0,>=1.14.9, but you have botocore 1.31.8 which is incompatible.
Successfully installed botocore-1.31.8 google-auth-2.22.0 urllib3-1.26.16

Alright. botocore's upgrade is incompatible with boto3, but otherwise no version conflicts. Excellent. Let’s catch that one too.

$ pip install --upgrade boto3

[... normal install stuff ...]

OK, no more dependency conflicts being reported. That’s a great sign. Maybe all I needed to do now was try upgrading all of these packages at once with pip-compile.

I looked at each of the pip install commands to see what all had been upgraded during this process, and I pulled all those dependencies into one pip-compile upgrade command. I’m used the short version of the --upgrade-package option, -P, for brevity. I found lines that looked like this, that told me all the packages that were installed.

Installing collected packages: urllib3, google-auth, botocore

And I crossed my fingers and ran the upgrade command.

$ pip-compile -P sentry-sdk -P urllib3 -P google-auth -P botocore -P boto -P s3transfer

No luck. It did an upgrade of one of the packages, but not sentry-sdk. Something looked off about s3transfer to me, I did’t understand why boto3 install listed it as installed, when all it actually did was remove it from the dependencies (you didn’t miss that, I cut it out of the logs). I wondered if perhaps specifying it as an upgrade package was causing pip-compile to make sure it was present in the lockfile, and that could be causing further issues. I removed it from the command and tried again.

$ pip-compile -P sentry-sdk -P urllib3 -P google-auth -P botocore -P boto3

Eureka! sentry-sdk and urllib3 were both upgraded to the latest versions in the lockfile, along with a few other package upgrades. I had found a minimal set of dependency upgrades to do in order to complete my task of upgrading sentry-sdk.

Final thoughts

At this point, I felt good. But I still needed to check over the version changelog of all of the packages that had been upgraded, to make sure none of them would cause me problems. But that’s the same old story as all upgrades, so I won’t bother you with a rundown of how that went. Suffice it to say, I was able to get the job done, and I’m happier for it!