Managing Dependencies in Python

pip, virtualenv, pipenv and poetry

Photo by chuttersnap on Unsplash

One of the things that makes Python developers really productive is the vast ecosystem of third party packages available to the developers.

But the experience of installing and maintaining packages for different projects is pretty fragmented. By default when we do pip install something the package is installed globally in your python environment.

The most conventional way to manage dependencies in python has been to install everything globally and then list all dependencies in a requirements.txt file, so that when someone clones your project then they can install all the dependencies using pip install -r requirements.txt

However, this method absolutely fails you when you have multiple projects and any two of the projects need different versions of the same package to work. Maybe one project is full of legacy code written with some older version of numpy, and you and your friend are working on something with the latest version.

“How do I install and manage different versions of the same package ?”

That’s where dependency management tools comes in. The python ecosystem has a lot of them, virtualenv, pipenv and poetry are the one’s used the most. Let’s have a look at all of them.

virtualenv

Installation : pip install virtualenv

virtualenv stands for Virtual Environments.

It helps you create an isolated python installation, so if you create a virtual environment for every project of yours, you have that many independent python installations and you can easily install all the packages for a project in each one of those without the them interfering with one another.

Once you have virtualenv installed, run the following command to create a virtual environment in the current location you’re in.

$ virtualenv my_project

This will create a directory named my_project which will contain the files for the isolated python installation

➜ ~ $ cd my_project
➜ ~/my_project/ $ ls
bin/  lib/  pyvenv.cfg

To access this newly created python installation,

➜ ~/my_project/ $ source bin/activate
(my_project) ➜  ~/my_project $ 

This will spawn a new shell, notice the (my_project), and this shell will give you access to this isolated python installation. You can navigate to your project source code in this shell to execute commands.

All the pip install commands you run in this shell will install the packages in that isolated python installation.

(my_project) ➜  ~/my_project $ pip install fastapi

This will install the “fastapi” package in the my_project virtual environment.

The standard practise usually is to

• Create a virtual environment for every project in some common directory where you store all the virtual environments for your projects
• Spawn a shell in that virtual environment
• Install dependencies using the requirements.txt
• Run all your python commands in the shell you just spawned as you need your code to access the packages in the virtual environment

This helps keep dependencies away from each other and now you are at liberty to choose specific package versions as per your project requirements.

Read more about the specific commands in the official docs here

pipenv

https://pipenv-fork.readthedocs.io/

The whole virtualenv setup is often a bit intimidating for beginners, and the fact that you have to manage the virtual environment directories is also a pain sometimes. Those are the problems pipenv tries to solve. The official python docs recommend pipenv for managing dependencies.

pipenv uses virtual environments and pip under the hood but offers a much simpler interface. It is very similar to npm which is the king of package managers in the javascript ecosystem.

Run pip install pipenv to install pipenv, you will now have the CLI installed.

To setup a project, just get into your empty project directory and run

➜ ~/my_project/ $ pipenv shell

running pipenv shell

• This will create a virtual environment with the same name as the parent directory.
• And you will have a file named Pipfile in your directory. This will automatically keep track of all the packages you install and their versions, kinda like requirements.txt

As we have not installed any packages in this environment yet, the contents of Pipfile will be as such

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
[requires]
python_version = "3.8"

Now you can install any package with the following command pipenv install <package name> , the command given below will install flask

(my_project) ➜  ~/my_project $ pipenv install flask

running pipenv install flask

Now the Pipfile will look like this

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packag/hellorerjvuyvdjhhg.rktkg
jjes]
[packages].............................................
flask = "*"
[requires]
python_version = "3.8"

When you clone or download a project with a Pipfile you can just run pipenv install to install all the dependencies and get the project going.

Some other useful commands

• pipenv install --dev to install the package as a dev dependency
• pipenv graph to get a tree like structure of the dependencies of your packages
• pipenv uninstall <package_name> to uninstall a package
• pipenv run <filename>.py if you are not in the virtual environment shell the you can run this command to directly run a file in the virtual environment.

Some useful articles regarding pipenv

Pipenv: A Guide to the New Python Packaging Tool - Real Python

Pipenv: Python Dev Workflow for Humans - pipenv 2020.6.2.dev0 documentation

poetry

Official Website of Poetry

Poetry

Yet another tool in the python ecosystem for managing dependencies is poetry. It’s has a lot more features and is a lot fancier compared to pipenv.

Installing Poetry

To install poetry in Linux, just paste the following in your terminal

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python

You can find os specific installation instructions here.

Using Poetry

To create a project in poetry, just run

$ poetry new my_project

This will generate a directory with a few files for you, essentially setting up a project boilerplate as such

.
├── my_project
│   └── __init__.py
├── pyproject.toml
├── README.rst
└── tests
    ├── __init__.py
    └── test_my_project.py
2 directories, 5 files

• The pyproject.toml file tracks your dependencies
• README.rst is for documentation
• All the source code and tests go in the my_project and tests folder respectively

The pyproject.toml file will be as such

[tool.poetry]
name = "my_project"
version = "0.1.0"
description = ""
authors = ["Junaid Rahim <junaidrahim8d@gmail.com>"]
[tool.poetry.dependencies]
python = "^3.8"
flask = "^1.1.2"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

You can also run poetry init . This will ask you a lot of questions in the command prompt and use those details to fill up the pyproject.toml file, but it will not give you a project structure with all the files and tests set up.

Installing Packages

To add packages, it’s as simple as

➜ ~/my_project/ $ poetry add flask

To start a shell in the created virtual environment, run

➜ ~/my_project/ $ poetry shell

Some other useful commands are

• poetry install to install the packages specified in pyproject.toml , useful when you clone a repo and want to set it up.
• poetry update will update all the packages to their latest versions
• poetry show will give you a list of all the installed packages
• poetry remove <package_name>will remove the specified package
• poetry run <command> will run the specified command in the virtual environment, for example poetry run main.py

You can read up in detail about all these commands here

Publishing Packages

Another area in which poetry really shines is publishing packages, if you are maintaining packages that are published on PyPI, usually it’s a lot of effort and configuration to publish a package but poetry handles it like a pro.

All you have to do is,

➜ ~/my_project/ $ poetry build

This will build all the wheels archives and then you can directly do

➜ ~/my_project/ $ poetry publish

It will automatically register the package before uploading if it is the first time it is submitted. And then it will publish the latest build.

Refer the official documentation for more details, it’s pretty neat.

Introduction | Documentation | Poetry - Python dependency management and packaging made easy.

There is also conda which is mostly used by the machine learning and data science folks, I wanted this list to be for any generic python project thus I did not go over that in detail, but you can read up about it on the link below

Conda - Conda documentation

I hope the next time you start a brand new python project you can pick a package manager that suits your requirements.

Thanks for reading. Drop a 👏 if you liked this article.