Poetry: Finally an all-in-one tool to manage Python packages

Dan Ettinger. © Niedermueller

Edit: Since poetry 1.2, there were some changes on how to install packages, I explained it here. For the rest, this tutorial is still valid.

If you are new to python programming, sometimes you feel overwhelmed when it comes to managing package dependencies and publishing your project on pypi or another form of package registry. There are too many tools involved in the ecosystem: virtualenv, pip, pipenv, flirt, twine, etc.. This is a bit frustrating because you don’t know what to use to begin and you very often have to juggle between several tools to get to your end.

Fortunately, a new tool has arrived and solved all these issues, it is called poetry! This tutorial will be based on poetry 1.0.0 released in December 2019.

Installation

On windows, you will need powershell to install it:

> (Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python

After that you will need to restart the shell to make it operational.

On linux and other posix systems (mac included):

$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python

To check that it is correctly installed, you can check the version:

> poetry --version
Poetry version 1.0.0

Here we see that we have version 1.0.0 installed. May be the latest stable version is not installed with the installation script, to update poetry, you can run:

> poetry self update

Please note that you can install poetry using the traditional pip command but poetry will be limited to create virtual environments for the python version for which it has been installed.

Initialize a project

To study poetry, we will create a simple project poet using the new command.

> poetry new poet

The tree structure of the created project looks like the following one.

├── pyproject.toml
├── README.rst
├── poet
│   └── __init__.py
└── tests
    ├── __init__.py
    └── test_poet.py

The most important file here is pyproject.toml which holds all the information necessary to manage our package. It looks like this at the beginning:

[tool.poetry]
name = "poet"
version = "0.1.0"
description = ""
authors = ["lewoudar <XXX@XXX.com>"]

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Here we have four sections defined:

• the first section defines all metadata not related to dependency packages like name, version, etc..
• The second section defines all the dependencies needed for the project
• The third section defines all development dependencies not necessary to build the project, but to perfom other actions like testing, building, documentation, etc..
• The fourth section defines a build system as indicated in PEP 517.

If you don’t like that poetry initializes a project for you or if you already have a project that you want to control with poetry, you can use the init command. You will get an interactive shell to configure your project.

> poetry init

Add a package

Ok let’s assume we want to use requests to perform some api queries, with poetry, add a dependency is straightforward.

> poetry add requests

If we want to install a development dependency, i.e not related directly with your project like pytest, we can do so by passing the -D option.

> poetry add -D pytest

N.B: If you previously used the new command, and add requests package like above, you probably already have pytest installed, so you will get an ValueError saying that the package already exists. So don’t be surprised :)

List dependencies

If we want to discover at some point all the installed dependencies, we can use the show command.

> poetry show --tree
pytest 5.3.2 pytest: simple powerful testing with Python
|-- atomicwrites >=1.0
|-- attrs >=17.4.0
|-- colorama *
|-- more-itertools >=4.0.0
|-- packaging *
|   |-- pyparsing >=2.0.2
|   `-- six *
|-- pluggy >=0.12,<1.0
|-- py >=1.5.0
`-- wcwidth *
requests 2.22.0 Python HTTP for Humans.
|-- certifi >=2017.4.17
|-- chardet >=3.0.2,<3.1.0
|-- idna >=2.5,<2.9
`-- urllib3 >=1.21.1,<1.25.0 || >1.25.0,<1.25.1 || >1.25.1,<1.26

The previous command draws a graph of all of our dependencies as well as the dependencies of our dependencies..

If we are not sure at some point that we have the latest version of a dependency, we can tell poetry to check on our package repository if there is a new version by using “— latest” option . We will get the packages listed with the current version and the latest released version.

> poetry show --latest
atomicwrites   1.3.0      1.3.0      Atomic file writes.
attrs          19.3.0     19.3.0     Classes Without Boilerplate
certifi        2019.11.28 2019.11.28 Python package for providing Mozilla's CA Bundle.
chardet        3.0.4      3.0.4      Universal encoding detector for Python 2 and 3
colorama       0.4.3      0.4.3      Cross-platform colored terminal text.
idna           2.8        2.8        Internationalized Domain Names 
...

Install the project

If we want to install the project in editable mode, we can just use the install command.

> poetry install

Note that if we have extra dependencies, they will not be installed with the previous command. Here is what the extra dependencies look like in the pyproject.toml file (taken from the official documentation):

[tool.poetry.dependencies]
# These packages are mandatory and form the core of this package’s distribution.
mandatory = "^1.0"

# A list of all of the optional dependencies, some of which are included in the
# below `extras`. They can be opted into by apps.
psycopg2 = { version = "^2.7", optional = true }
mysqlclient = { version = "^1.3", optional = true }

[tool.poetry.extras]
mysql = ["mysqlclient"]
pgsql = ["psycopg2"]

So if we want to install the project plus the postgresql client, this is what we should enter in the terminal:

> poetry install -E pgsql

And if we want the all the database clients:

> poetry install -E mysql -E pgsql

Handle virtual environment

When we use poetry, the virtual environment is created automatically somewhere on our system. The exact place depends on the OS we have, but we can find it running the env command and looking for the path info.

> poetry env info
Virtualenv
Python:         3.8.0
Implementation: CPython
Path:           C:\Users\rolla\AppData\Local\pypoetry\Cache\virtualenvs\poet-6mnhVs-j-py3.8
Valid:          True
System
Platform: win32
OS:       nt
Python:   C:\Users\rolla\AppData\Local\Programs\Python\Python38-32

In fact we can add “ — path” option to the previous command to only get the path of the virtual environment ;)

> poetry env info --path
C:\Users\rolla\AppData\Local\pypoetry\Cache\virtualenvs\poet-6mnhVs-j-py3.8

If we want to run a script installed by one of our dependencies, we need a way to access our virtual environment otherwise our OS will complain that it doesn’t know the command. To do that poetry provides a run command to solve this issue. It will automatically run the arguments passed to it in the virtual environment.

> poetry run pytest
===================================================================================== test session starts =====================================================================================
platform win32 -- Python 3.8.0, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: D:\projets\python\tutorials\poet
collected 1 item
tests\test_poet.py .                                                                                                                                                                     [100%]
..

We can also, for example, uninstall the project in editable mode by running the pip command.

> poetry run pip uninstall poet

If we find annoying to always add a poetry run in front of all the script commands we want to use, poetry provides a shell command that spawns a new shell directly inside the virtual environment. With that we will now be able to call pytest command without poetry run in front of it.

> poetry shell
> pytest
===================================================================================== test session starts =====================================================================================
platform win32 -- Python 3.8.0, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
...

To leave this shell, we can just run exit.

You can have a look at this documentation if you want to manage multiple python interpreters.

Update packages

To update all dependencies we can run the update command.

> poetry update

If we just want to update some packages, we can specify them as arguments of the update command.

> poetry update package1 package2

Remove a package

Remove a package is also straightforward, thanks to the remove command.

> poetry remove requests

If it is a development package we must pass the -D option to the command.

> poetry remove -D pytest

Build the final package

Add some useful metadata to pyproject.toml

Before building the project, we need to add some useful metadata to distribute and explain how to use our package like the license, some keywords and the packages we want to include in the final build. Our pyproject will now look like this:

[tool.poetry]
name = "poet"
version = "0.1.0"
description = ""
authors = ["lewoudar <XXX@XXX.com>"]
license  = "Apache-2"
keywords = ["poet", "requests"]
readme = "README.rst"

packages = [
    { include = "poet" }
]

[tool.poetry.dependencies]
python = "^3.8"
requests = "^2.22.0"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

If you want to know more about the different metadata to pass to pyproject.toml, look at this documentation.

Build

To build a package, we run the.. build command!

> poetry build
Building poet (0.1.0)
 - Building sdist
 - Built poet-0.1.0.tar.gz
- Building wheel
 - Built poet-0.1.0-py3-none-any.whl

If we want to limit the build to a specific type, we can just use the -F option.

> poetry build -F wheel # or sdist

Deploy a package

Configure repositories and credentials

If we are not deploying on the traditional pypi we should configure the repository where we want to upload our package. The config command is created for this purpose.

> poetry config repositories.priv https://private.repository

In the above example we configure the repository named “priv” with the url “https://private.repository”.

If we want to store credentials for this private repository, we can also do it using the config command.

poetry config http-basic.priv username password

In the above command, “username” and “password” corresponds to our.. username and password to login to our private repository. Note that if we don’t specify the password, we will be prompted to write it when we will deploy our package.

For pypi, we can also configure credentials using the previous command and replacing “priv” by “pypi” but it is now recommended to use API tokens to authenticate with pypi. We can configure it like this:

poetry config pypi-token.pypi my-token

“my-token” represents the token we created on pypi.

Note that we can also use environment variables to configure our credentials. This is particularly useful when using a CI tool.

Deploy

When we are done with configuration, we can simply deploy our package using the publish command.

poetry publish

If we used a private repository, we must specify the name we previously configured via the -r option.

poetry publish -r priv

Note that we can use the “ — build” option to build the package before deploying it.

C Extensions

Currently support for C extensions is not fully operational. You can get a draft solution by reading this issue, but I will advice you to continue using standard tools like pip or pipenv unless poetry fully supports it.

That’s all for this tutorial. Hope you enjoyed it and learned a lot :)