Python dependency management — timeline overview

Introduction

This article’s intent is to describe the evolution of python packaging and python dependency management, but since this is a ‘modern’ history I won’t allocate much of your time on ‘history’. Instead I will rather guide you through the “evolution”, focusing more on the recent methods of managing dependencies in python.

The rise of python packaging

Although python started around 1990s, it took quite long for it to adapt to patterns that other programming languages were already using when distributing software, such as Perl with CPAN.

Before 2000, in order to distribute your program, you practically had to upload it somewhere on the internet where you had to specify exact instructions of installing the software to the python interpreter. Thus, the community started distutils. That introduced the files setup.py, setup.cfg which contained instructions of how you should install the software and other useful metadata.

The dependency chaos

The python community needed something more than distutils, therefore setuptools was created more like a wrapper over the standard library. I won’t go in details of what offered over the standard library. One key important feature was easy_install which allowed you to install other packages to your python interpreter.

Although in order to install a package you needed root access, which didn’t take long to turn into chaos as the python community began to split, some using distutils and some setuptools. Regardless of this, all packages would built into one place bringing up the issue with root access. Introducing confusion into the deployment process and diverted from some of the standard security principles such as never run anything as root.

The need for isolated environment arose, and around 2007 virtualenv emerged. This concept allowed developers to have multiple python interpreters per each project, isolating a python interpreter from the base installation of python. With this, dependency management has improved giving multiple benefits such as environment exports and environment isolation.

Pip was introduced and replaced the quirky easy_install, pip also introduced the concept of requirements.txt setting a standard inside the python community. The file could be generated by extracting all the dependencies from the current virtualenv. Later making possible for pip to read the file and recreate the same virtual environment.

e.g

pip freeze > requirements.txt 
pip install -r requirements.txt

The cool feature that pip provided, is that now you are able to uninstall a package. Which the python community has not heard of such ability until pip arrival, despite the fact that it took python community 10 years to provide this feature.

The big issue with virtualenv

The creation of virtualenv had to be managed by the developer. Although the environment had to be managed via pip and other tools, making the entire process more complex than it should be. Besides, managing requirements.txt became problematic as it was unable to produce deterministic builds for various reasons.

e.g

virtualenv env 
source env/bin/activate
pip freeze > requirements.txt 
pip install -r requirements.txt

The Renaissance of Python dependency management

Hitherto pip and virtualenv bloated the developers as they had to use two separate tools to isolate their environment and manage their dependencies. Despite the fact that npm and composer already had a solution for this issue, python community learned from other programming languages and were able to introduce projects such as pipenv.

Pipenv introduced two new files to the python stack, Pipfile which replaced requirements.txt and the newly Pipfile.lock which ‘locked’ your dependencies, allowing you the produce deterministic builds. In addition, pipenv managed your virtualenvs, by automating environment creation, environment exports, and allowed you to even run a shell inside that environment.

the command below will automatically create a virtual environment and spawn a shell inside it, it will also create Pipfile and Pipfile.lock.

pipenv shell

Even though pipenv was awesome for developers that develop applications, but it didn’t help so much the libraries. Now you have to handle even more files Pipfile, Pipfile.lock, setup.py, etc…

Thus it didn’t took too long from Pipenv arrival for Poetry to show off his abilities by taking inspiration from tools like composer. Poetry does everything what Pipenv does but its key feature is that now everything is merged into one file `pyproject.toml`, removing the overhead of managing multiple files.

Poetry is a full-fledged package manager, that offers more than just dependency management and packaging. It also tries to enforce standards such as semantic versioning, folder structure, and packaging patterns that programmers from other communities are most probably familiar with.

As a conclusion, I would describe pipenv as perfect for applications and poetry ideal for libraries, although this will again introduce confusion and separation inside the python community. Therefore, considering poetry advantages over pipenv, in my perspective, it would be ideal for poetry to standardise inside the python community.