Read the original article on my blog

New Year 2020 marks the end of more than a decade of coexistence of Python 2 and 3. The Python landscape has changed considerably over this period: a host of new tools and best practices now improve the Python developer experience. Their adoption, however, lags behind due to the constraints of legacy support.

This article series is a guide to modern Python tooling with a focus on simplicity and minimalism.¹ It walks you through the creation of a complete and up-to-date Python project structure, with unit tests, static analysis, type-checking, documentation, and continuous integration and delivery.

This guide is aimed at beginners who are keen to learn best practises from the start, and seasoned Python developers whose workflows are affected by boilerplate and workarounds required by the legacy toolbox.

Requirements

You need a recent Linux, Unix, or Mac system with bash, curl and git for this tutorial.

On Windows 10, enable the Windows Subsystem for Linux (WSL) and install the Ubuntu 18.04 LTS distribution. Open Ubuntu from the Start Menu, and install additional packages using the following command:

Overview

In this first chapter, we set up a Python project using pyenv and Poetry. Our example project is a simple command-line application, which uses the Wikipedia API to display random facts on the console.

Here are the topics covered in this chapter:

Here is a list of the articles in this series:

This guide has a companion repository: cjolowicz/hypermodern-python. Each article in the guide corresponds to a set of commits in the GitHub repository:

Setting up a GitHub repository

For the purposes of this guide, GitHub is used to host the public git repository for your project. Other popular options are GitLab and BitBucket. Create a repository, and populate it with README.md and LICENSE files. For this project, I will use the MIT license, a simple permissive license.

Throughout this guide, replace hypermodern-python with the name of your own repository. Choose a different name to avoid a name collision on PyPI.

Clone the repository to your machine, and cd into it:

As you follow the rest of this guide, create a series of small, atomic commits documenting your steps. Use git status to discover files generated by commands shown in the guide.

Installing Python with pyenv

Let’s continue by setting up the developer environment. First you need to get a recent Python. Don’t bother with package managers or official binaries. The tool of choice is pyenv, a Python version manager. Install it like this:

Add the following lines to your ~/.bashrc:

Open a new shell, or source ~/.bashrc in your current shell:

Install the Python build dependencies for your platform, using one of the commands listed in the official instructions. For example, on a recent Ubuntu this would be:

You’re ready to install the latest Python releases. This may take a while:

Make your fresh Pythons available inside the repository:

Congratulations! You have access to the latest and greatest of Python:

$ python3.7 --version
Python 3.7.7

Python 3.8.2 is the default version and can be invoked as python, but both versions are accessible as python3.7 and python3.8, respectively.

Setting up a Python project using Poetry

Poetry is a tool to manage Python packaging and dependencies. Its ease of use and support for modern workflows make it the ideal successor to the venerable setuptools. It is similar to npm and yarn in the JavaScript world, and to other modern package and dependency managers. For alternatives to Poetry, have a look at flit, pipenv, pyflow, and dephell.

Install Poetry:

Open a new login shell or source ~/.poetry/env in your current shell:

Initialize your Python project using poetry init:

This command will create a pyproject.toml file, the new Python package configuration file specified in PEP 517 and 518.

[tool.poetry.dependencies]
python = "^3.8"
[tool.poetry.dev-dependencies][build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

There you go: one declarative file in TOML syntax, containing the entire package configuration. Let’s add some metadata to the package:

Poetry added a dependency on Python 3.8, because this is the Python version you ran it in. Support the previous release as well by changing this to Python 3.7:

The caret (^) in front of the version number means “up to the next major release”. In other words, you are promising that your package won't break when users upgrade to Python 3.8 or 3.9, but you're giving no guarantees for its use with a future Python 4.0.

Creating a package in src layout

Let’s create an initial skeleton package. Organize your package in src layout, like this:

2 directories, 2 files

The source file contains only a version declaration:

Use snake case for the package name hypermodern_python, as opposed to the kebab case used for the repository name hypermodern-python. In other words, name the package after your repository, replacing hyphens by underscores.

Replace hypermodern-python with the name of your own repository, to avoid a name collision on PyPI.

Managing virtual environments with Poetry

A virtual environment gives your project an isolated runtime environment, consisting of a specific Python version and an independent set of installed Python packages. This way, the dependencies of your current project don’t interfere with the system-wide Python installation, or other projects you’re working on.

Poetry manages virtual environments for your projects. To see it in action, install the skeleton package using poetry install:

Creating virtualenv hypermodern-python-rLESuZJY-py3.8 in …/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (0.1s)
Writing lock fileNothing to install or update - Installing hypermodern-python (0.1.0)

Poetry has now created a virtual environment dedicated to your project, and installed your initial package into it. It has also created a so-called lock file, named poetry.lock. You will learn more about this file in the next section.

Let’s run a Python session inside the new virtual environment, using poetry run:

Python 3.8.2 (default, Feb 26 2020, 07:03:58)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hypermodern_python
>>> hypermodern_python.__version__
'0.1.0'
>>>

Managing dependencies with Poetry

Let’s install the first dependency, the click package. This Python package allows you to create beautiful command-line interfaces in a composable way with as little code as necessary. You can install dependencies using poetry add:

Using version ^7.0 for clickUpdating dependencies
Resolving dependencies... (0.1s)
Writing lock file
Package operations: 1 install, 0 updates, 0 removals - Installing click (7.0)

Several things are happening here:

  • The package is downloaded and installed into the virtual environment.
  • The installed version is registered in the lock file poetry.lock.
  • A more general version constraint is added to pyproject.toml.

The dependency entry in pyproject.toml contains a version constraint for the installed package: ^7.0. This means that users of the package need to have at least the current release, 7.0. The constraint also allows newer releases of the package, as long as the version number does not indicate breaking changes. (After 1.0.0, Semantic Versioning limits breaking changes to major releases.)

By contrast, poetry.lock contains the exact version of click installed into the virtual environment. Place this file under source control. It allows everybody in your team to work with the same environment. It also helps you keep production and development environments as similar as possible.

Upgrading the dependency to a new minor or patch release is now as easy as invoking poetry update with the package name:

To upgrade to a new major release, you need to update the version constraint explicitly. Coming from the previous major release of click, you could use the following command to upgrade to 7.0:

Command-line interfaces with click

Time to add some actual code to the package. As you may have guessed, we’re going to create a console application using click:

from . import __version__
@click.command()
@click.version_option(version=__version__)
def main():
"""The hypermodern Python project."""
click.echo("Hello, world!")

The console module defines a minimal command-line application, supporting --help and --version options.

Register the script in pyproject.toml:

Finally, install the package into the virtual environment:

You can now run the script like this:

Hello, world!

You can also pass options to your script:

Usage: hypermodern-python [OPTIONS]  The hypermodern Python project.Options:
--version Show the version and exit.
--help Show this message and exit.

Example: Consuming a REST API with requests

Let’s build an example application which prints random facts to the console. The data is retrieved from the Wikipedia API.

Install the requests package, the de facto standard for making HTTP requests in Python:

Next, replace the file src/hypermodern-python/console.py with the source code shown below.

import click
import requests
from . import __version__
API_URL = "https://en.wikipedia.org/api/rest_v1/page/random/summary"
@click.command()
@click.version_option(version=__version__)
def main():
"""The hypermodern Python project."""
with requests.get(API_URL) as response:
response.raise_for_status()
data = response.json()
title = data["title"]
extract = data["extract"]
click.secho(title, fg="green")
click.echo(textwrap.fill(extract))

Let’s have a look at the imports at the top of the module first.

import click
import requests
from . import __version__

The textwrap module from the standard library allows you to wrap lines when printing text to the console. We also import the newly installed requests package. Blank lines serve to group imports as recommended in PEP 8 (standard library–third party packages–local imports).

The API_URL constant points to the REST API of the English Wikipedia, or more specifically, its /page/random/summary endpoint, which returns the summary of a random Wikipedia article.

In the body of the main function, the requests.get invocation sends an HTTP GET request to the Wikipedia API. The with statement ensures that the HTTP connection is closed at the end of the block. Before looking at the response body, we check the HTTP status code and raise an exception if it signals an error. The response body contains the resource data in JSON format, which can be accessed using the response.json() method.

We are only interested in the title and extract attributes, containing the title of the Wikipedia page and a short plain text extract, respectively.

Finally, we print the title and extract to the console, using the click.echo and click.secho functions. The latter function allows you to specify the foreground color using the fg keyword attribute. The textwrap.fill function wraps the text in extract so that every line is at most 70 characters long.

Let’s try it out!

Jägersbleeker Teich
The Jägersbleeker Teich in the Harz Mountains of central Germany
is a storage pond near the town of Clausthal-Zellerfeld in the
county of Goslar in Lower Saxony. It is one of the Upper Harz Ponds
that were created for the mining industry.

Feel free to play around with this a little. Here are some things you might want to try:

  • Display a friendly error message when the API is not reachable.
  • Add an option to select the Wikipedia edition for another language.
  • If you feel adventurous: auto-detect the user’s preferred language edition, using locale.

Thanks for reading!

The next chapter is about adding unit tests to your project.

Continue to the next chapter
  1. The title of this guide is inspired by the book Die hypermoderne Schachpartie (The hypermodern chess game), written by Savielly Tartakower in 1924. It surveys the revolution that had taken place in chess theory in the decade after the First World War. The images in this chapter are details from the hand-colored print La Sortie de l’opéra en l’an 2000 (Leaving the opera in the year 2000) by Albert Robida, ca 1902 (source: Library of Congress).

Berlin-based software author and musician. https://cjolowicz.github.io/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store