RIP Pipenv: Tried Too Hard. Do what you need with pip-tools.

Nick Timkovich
Jan 24, 2020 · 4 min read

Pipenv is dead. It went all of 2019 without a single release, despite about 650 commits to master since the last release. Comments from developers on the project explain it “has been held back by several subdependencies and a complicated release process”. In the past, there were also comments that internal changes to pip broke it, but pip is continuing to evolve in response to the PSF funding developers to work on it. Will they hit this moving target and make a release in 2020? Do I care?

Rocketing in popularity! (Image: NASA/Joel Kowsky)

My principal reasons for using Pipenv were twofold. First, was the ability to have reproducible builds through an exhaustively defined lock file, with every package and version specified, then hashed for good measure. Second, I could do the above while maintaining a clean, minimal list of direct dependencies for my application.

Pin everything…

The reason for a lock file should be clear: so you can have reproducible builds. may work when you build and test on your workstation, and through one release cycle, but what about when you make a small fix later? Why did changing one line of code and rebuilding cause everything to explode? There was a new incompatible release among one of your dependencies. Precisely defining all packages and their exact versions leads to reproducible builds that are easier to diagnose and bisect when they go wrong.

Bonus features in the lock file include package hashes. It can improve security, but the failures I’ve seen are because of using multiple package indexes. At Telnyx we have a private PyPI that we use to distribute Python libraries. We adapted a synchronous library to asyncio, but later someone else did so and released it publically with the same name. The collision led to sometimes downloading the private version, sometimes the public, depending on how the indexes were configured on the user’s machine, their build container, or the CI infrastructure. Our private PyPI also allows re-uploading of the same version, which is also made clear when the hash check fails.

The locked list of dependencies should be deliberately generated and validated through tests. This is often by a developer on their workstation, and they can immediately check before entering a broken commit into the repo. Diffing the lock file will show what new versions were installed, and the version can be pinned to an older version, or source changed to accommodate the change.

…and also only what you need

OK, but why do we need a separate list of dependencies? Imagine a simple cycle of pinning everything in a singular and regenerating it with . The file will not track why certain packages are installed, and if you explicitly remove a direct dependency from it, you’ll be keeping around all of its sub-dependencies for no reason. Additionally, you can enter version resolution hell wherein trying to bump one package for a new version means it needs another package, but a different version than you have pinned. Why is that package pinned? Are you using it directly where you care about the changes, or is it just for one of your deps? Or multiple?

In this minimal list, we can pin just what we need, excluding incompatible or buggy versions purposefully, and allowing incidental upgrades where we can. This can provide bug fixes and performance improvements (admittedly there can be regressions…), and preclude integrating with deprecated features in older libraries that will be a large maintenance burden later.

pip-compile: does one thing

Note that none of the features listed include virtual environment management. I use Pyenv for this on my machine, and in production, setting using pip and the built-in package is best. Why install anything when you can create a virtual environment in one line, and pip install with one more? They’re ever-so-slightly onerous to call if you use full paths, but you’re typing it once in the Dockerfile or other setup scripts anyways.

So how do I get that while holding on to the features I liked? Enter pip-tools. Making a wiser decision than Pipenv, pip-tools goes with the UNIX philosophy of “do one thing, and do it well”. It provides a few commands, but one of them: can bake one requirements-style file into another, complete with versions and hashes.

Pipenv uses this library to do it — we can cut out the middleman! What’s more, the generated is perfectly readable by pip, so you don’t need to install the tooling in your production system, saving time, space, and exposure to bugs.

I think I’ve been guilty of inflating Pipenv’s download stats; every container build pulls it

How can you make the switch? If you’re willing to shake up the versions from your , converting the Pipfile to a is sometimes just a slightly-tedious bunch of editing. However, if you have a lot of libraries, and want to keep their versions locked to make sure changing tools didn’t break anything, I wrote a script that will do it for you!

So even if Pipenv came out with a new release fixing all the outstanding bugs and pain points I’ve come across, I won’t go back.

Telnyx Engineering

Stories by Engineers for Engineers

Thanks to Telnyx Engineering

Nick Timkovich

Written by

Software Engineer at Telnyx | ChiPy regular, Having fun with Go | Gamer, Tinkerer, Baker, Hot Sauce Maker | now: Chicago, then: Grand Rapids, Michigan

Telnyx Engineering

Stories by Engineers for Engineers

Nick Timkovich

Written by

Software Engineer at Telnyx | ChiPy regular, Having fun with Go | Gamer, Tinkerer, Baker, Hot Sauce Maker | now: Chicago, then: Grand Rapids, Michigan

Telnyx Engineering

Stories by Engineers for Engineers

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store