Python Tooling Makes a Project Tick

An opinionated guide to tooling in Python covering pyenv, poetry, black, flake8, isort, pre-commit, pytest, coverage, tox, Azure Pipelines, sphinx, and readthedocs

Adithya Balaji

Here’s a link to the TL;DR

Introduction

So you’ve got an awesome idea for a new open-source package that’s going to rock the Python landscape. You might be wondering how you make sure you set up the package so that the open-source community can help take your idea to the next level. Or, you might be wondering how can you solve the problems you’re running into when setting a personal project. For example, you might be running into merge conflicts due to inconsistent styles or broken refactors because the appropriate tests are not in place. This blog will help you here as well!

Tooling is the answer! This interactive guide will help you set up your project (or level up your skills) with the current Python best practices making sure that you can focus on the awesome idea you have and not worry about racking up technical debt with code that’s hard to maintain. Although we cover quite a few tools, you will see a lot of them are bundled in one or two commands so you can run all of them at the same time.

What can I get out of this?

Software Engineer: You’ll find some new packages to make it easier to code in Python.

DevOps Engineer: You’ll get an introduction on how to make your software engineer’s lives easier with Python-centric tooling and a contextual introduction to Azure Pipelines.

Data Scientist: You will find a collection of code formatting tools that will help when writing machine learning scripts.

List of Tools Covered

Environment

• pyenv
• pyenv-virtualenv
• poetry

Project Styling

• flake8
• flake8-docstrings
• darglint
• black
• isort
• seed-isort-config
• pre-commit

Unit Testing

• pytest
• pytest-cov
• pytest-mock
• xdoctest
• coverage
• tox

Continuous Integration

• Azure Pipelines

Documentation

• sphinx
• readthedocs
• sphinx_rtd_theme

Release

• towncrier
• poetry (again)

Setting up your environment

Pyenv

Before you even get started with your project, it’s important to get your Python version setup right. For reasons that will become evident in later sections, it may be very useful to have multiple Python versions installed on the same machine. Pyenv helps us handle that issue.

The installer provides a few tools: pyenv, pyenv-doctor, pyenv-installer, pyenv-update, pyenv-virtualenv, and pyenv-which-ext. Below, we’ll cover the main tool pyenv and pyen-virtualenv.

After the initial installation, you’ll see some commands that install multiple versions of Python on your system in addition to the system Python version installed on most machines (on MacOS it’s Python 2.7.10). Don’t worry this won’t mess with your current setup as they are isolated from the rest of your setup. Look out for the pyenv global which sets a global python version to use everywhere. At any point, you can always set it back to your system Python version by using pyenv global system.

Pyenv allows you to set up specific versions of python that get activated in a specific directory. This is achieved through the pyenv local [PY_VERSION]command. This creates .python-version files in directories that automatically point the python command to the correct version of Python for your project. If you are interested in how this works, look into the concept of shims explained in the pyenv docs.

This also allows you to use pyenv to set up multiple Python versions in a directory for testing purposes (covered in the tox section). The killer feature here is a part of the pyenv-virtualenv tool which, when set up, allows you to automatically activate and deactivate Python virtual environments based on your directory (if you set the env as local). Pretty handy, huh?

Note if you are using a shell that isn’t the default bash shell you will need to simply apply steps 2 and 3 in the Basic GitHub Checkout install instructions of pyenv. This adds pyenv to your path and automatically activates the tool upon shell initialization. Same goes for pyenv-virtualenv step 2 which also needs to be completed for non-bash users. Another exciting thing to note is that pyenv works with conda which means you can move your entire python management system to pyenv.

Poetry

Next, you need to install a key player in your Python package tooling system: poetry. Quoting the poetry documentation: “Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right stack everywhere.” Poetry ensures all your packages play nicely with each other and is the way forward following PEP517. Poetry uses a centralized file called pyproject.toml to centralize all your package configurations alongside its dependencies.

If you’re wondering what a toml file even is, it’s just another file format like JSON, INI, and YAML. The PEP517 authors selected this as the best format for encoding this information.

This is the best way to get poetry set up on your machine:

This temporarily sticky sets our Python version to the system Python. You then install a version of poetry on your computer with its dependencies isolated from the rest of the system. This means that poetry’s dependencies don’t get in the way of your day-to-day Python projects’ dependencies.

Note, once again if you are using zsh or another non-standard shell you will need to manually perform the following steps as documented in the poetry documentation. Essentially, you need to add source “$HOME/.poetry/env” to your shell rc file.

Now you need to clone the base project. Note that you are cloning a pre-configured branch that you will use to experiment with the tools covered here. If you want to see a clean project take a look at the master branch: git checkout master.

Now let’s set up a virtualenv to store the dependencies of this project.

Let’s recap. We moved into the directory of the project and created a virtualenv which gets stored in the root pyenv directory (under a ./versions subdirectory) keeping our project directory clean. Note that virtualenvs use the default version of python that is active in the directory if not specified. For example, this could be the global version that we set in previous steps. (Python 3.6.8)

We then set the local Python version to simplecalc, the virtualenv, with 3.6.8 and 3.7.3 being other options as well. Now you might say, “hold on, how can there be multiple versions in a single directory?” Well, the beauty of pyenv is that it allows you to use multiple shimmed “python versions” that get added to your PATH per directory.

These specific commands, such as python3.6 and python3.7, can be used by tools such as tox when testing multiple python versions. It is important to note that, regardless, when you type python the version is mapped to the first one in the local list you specified. In this case, that is the “convenience virtualenv”, simplecalc. Also, note that pyenv-virtualenv also automatically activates and deactivates the virtualenv as you cd in and cd out of the directory.

Now, simply install the project (poetry by default makes sure to install development requirements, specified in pyproject.toml, when you are developing locally). Note these are only installed to this presently-activated virtualenv (simplecalc) for use when developing. When testing, tox will create additional isolated environments using those additional python versions you added to your PATH. Now you are ready to walk through the rest of the tooling in the project.

An important concept to understand is the difference between the pyproject.toml and poetry.lock file. The pyproject.toml file encodes the semantic version specifications for your dependencies and the poetry.lock file encodes the exact specifications of your project. In other words, think of poetry.lock as nearly equivalent to file version of the output of a (f) a pip freeze command whereas as the pyproject.toml specifies your tolerances for updates. For example, click = “⁷.0” which translates to pip speak as >=7.0 <8.0.0. For more information on how this works, check out this page in the poetry docs.

If you run poetry lock Poetry will resolve the specifications set in the pyproject.toml to the latest packages that have been released and add those to your poetry.lock file. If you run poetry update the packages will both lock and update in your virtualenv. Think of update as a lock followed by an install that also allows you to specify a specific package to target.

You should probably put your poetry.lock file in a Version Control System (VCS) like git if you want to make sure that all cloners of your project install the same dependencies. One other thing to note is that poetry plays nice with pip as you can install a repo with pip install. Lastly, at the end of this guide, you’ll see that poetry can also be used to package your project as a replacement for twine.

See this gist if you’d like to see the series of commands that were run to build up the pyproject.toml file.

Unit Testing

The most important part of any good programming project is Unit Testing. Unit Testing and Continuous Integration (CI), covered later, go hand in hand as testing ensures all parts of your code work as intended and CI ensures that those tests are run automatically every time you push a change to your code upstream. Together they allow a system to cope with the stresses of a multitude of open-source contributors working on the same project and let you verify code before it is merged into your main branches. There are already many blogs that go in depth into why unit testing is useful.

pytest, pytest-cov, pytest-mock, and xdoctest

pytest is a unit testing platform and the package of choice when it comes to the majority of the Python ecosystem. It has quite the feature set when compared against the default Python unit test package. The tool automatically detects your testing directory and collects the tests you have written to be executed. When executing the tests it provides a convenient debugging interface where you can simply provide the — pdb flag to hunt down errors in failing tests. As with many other good packages, it supports an extensive plugin ecosystem which allows for additional testing functionality. The three packages we recommend are pytest-cov, pytest-mock and xdoctest. pytest-cov integrates with the coverage command which we explain in the next section. pytest-mock allows you to more easily access Python’s mocking functionality without many nested context managers (with statements). If you aren’t familiar with how mocks work, here is a great stackoverflow answer to get started on the topic. xdoctest is a great package that builds on the Python doctest package, providing cleaner Google-style doc integration and directly interfaces with pytest. All of these packages can be configured through flake8’s command-line flags and also through the setup.cfg file.

coverage

While tests are great, it’s just as important to measure your testing as it is to write them. The best way to do that is to use the Python coverage package. This package takes a pytest run and interprets the execution to determine which lines of code were run and which weren’t. This gives a development team a rough estimate of the areas that need greater attention when developing a set of tests. While there are a few different ways to measure coverage, the most robust method is branch coverage which looks at all of the diverging pathways that your code can take, making sure your tests hit each of those cases. When pytest-cov package is installed, this tool runs automatically following your tests to give you an idea of where you stand. To make sure that coverage is set up correctly see an example setup.cfg file.

Example

Let’s pause to see how these two tools work in practice. First, let’s try to run pytest and see what we get.

Running pytest runs all of the plugins that you installed including xdoctest and coverage. There are a few bugs that you need to patch to understand how pytest works. You can find the bugs yourself or simply use the specific commit git diff listed in the gist to see the bugs we introduced to know what needs to be fixed.

An astute observer might note that the coverage is less than 100%. Let’s fix that. To make your life easy, the tests are already written but just commented out.

Once fixed, the coverage should be back up to 100%, don’t worry you don’t always have to have your coverage that high it’s just a guide for sections of your code that need your attention.

tox

tox is a powerful Python testing automation package. It automates the setup and execution of the testing steps above. You can use it to test across multiple Python versions. tox plays well with pyenv and, if you specify the “matrix” of tests to run it builds an isolated virtual environment where your package is installed and tests are run. This brings some of the convenience of CI system debugging right back into your native machine. You can configure it directly in pyproject.toml. It is important to note that tox builds isolated virtualenvs inside a .tox folder where your project is installed and tested against the versions specified by the envlist ini field. (py36 and py37 are tox shorthands to specify the python version to be used).

Note that running pytest, as done above, only runs on the version of python that the virtualenv used (a proxy for testing your code across all versions). In order to debug issues in other python versions, you would use tox.

Example

Let’s see how it works on our example project: Since the output is quite large, we’ve linked the gist showing how to run tox here instead of embedding it.

Project Styling

Another element of a project you should consider with any Python project is the readability of the code. An important component of readability is the consistent coding style used by the project.

flake8, flake8-docstrings, darglint

Flake8 is a fantastic package and tool that will make sure that your code is in tip-top shape. The package follows the “letter of the law” for Python styling as defined by the PEP8 guidelines. Flake8 has two default backends it uses to check your code: pycodestyle and pyflakes. Both are maintained by PyCQA (Python Code Quality Authority) which, like PyPA (maintainers of pip), manages many Python code styling related packages. Pycodestyle checks the code for PEP8 compliance and pyflakes checks the code for common errors and linting mistakes like unused imports or variables. Flake8 can also keep your code from getting too complex (and unmaintainable) through the use of a McCabe check which is a fantastic summary statistic of code line complexity. Lastly, flake8 can be configured via setup.cfg (they are adding pyproject.toml support).

Flake8 also supports a rich ecosystem of plugins. Packages that we recommend include flake8-docstrings and darglint which both parse your code and flag issues as part of the flake8 command. Both support Google docstrings which are both readable in code and can easily turn into an HTML form. flake8-docstrings has a back-end of pydocstyle, also maintained by PyCQA, which checks you code docstrings for errors and nudges you to add them where needed. `darglint` protects you from changing your function signature without updating the API docstrings.

Let’s see the issues flake8 identifies with our example branch.

We’ll resolve these issues at the end.

black

With all of those linting (checking code for possible errors) and styling demands you might be wondering: “Is there something that can do that automatically?” Well, funny you should ask because there’s a tool just for that: black.

“You can have any color as long as it’s black.”
Henry Ford (1909)

This tool provides an opinionated set of code style guidelines to follow and implements the tooling to make those changes as well. Simply running black on your code will automatically fix most of those errors. (You might still need to manually fix your docstrings as unfortunately, it doesn’t have telepathic power). The tool quickly gained immense community traction and is now in fact under the python software foundation umbrella. You can configure some of the project settings in the pyproject.toml.

Let’s see all the files that black would modify. Again, we’ll fix these later.

isort and seed-isort-config

Another very useful tool is isort. This tool helps make sure that your imports are in the correct order as per the PEP8 guidelines and automatically sorts them to match that style if they aren’t. They are grouped into standard, external, local, and alphabetized. The style of imports isort opts for has the added bonus of reducing merge conflicts as well. In order to automatically recognize external packages, another tool called seed-isort-config jumps in to automatically seed the configuration step in the pyproject.toml file.

Well, what does isort suggest we change in our example?

pre-commit

Lastly, you might be thinking that this slew of tools is a lot to remember for every piece of code you commit. Never fear, another tool can help: pre-commit As the name implies, pre-commit is a Python package that allows you to create a .pre-commit-config.yaml file that maintains a list of tools to run before each commit. It runs only on the files that you have staged for commit. Here’s how you set it up in an existing repo:

pre-commit will only run on the staged file prior to a commit. Sometimes, you might want to run the steps specified in pre-commit without having to commit anything. pre-commit run — all-files, as shown in the example, achieves this. Other times, you just want to commit broken or temporary code and that’s fine. You can do that by adding the — no-verify flag to a commit. (ie: git commit -m “MSG” — no-verify)

Let’s now go through and fix the flake8 issues to see a working pre-commit run.

We now have a painless process to apply the linting changes that are needed for the project. black and isort will do most of the heavy lifting but generally, you will still need to manually address the docstring issues and other fixes the tools can’t automatically do.

Usually, you won’t have to do all of those steps. We did here for pedagogical purposes. The workflow you should follow is to run git commit -m “MSG” and the system will automatically catch the errors for you. You simply need to stage the changes and address flake8.

Now just commit the changes to check that all is well.

Continuous Integration (CI)

As we mentioned earlier, continuous integration is the second piece of the project maintenance puzzle. It allows you to run your tests each time you merge code and prevent one bad merged commit from derailing a project release. You’ve probably had the frustrating experience of trying to track down a bug. CI paired with well-written tests can help with that as you get a friendly green checkmark on GitHub for each commit that passed CI. There are already a ton of blogs that show why continuous integration is great and I’ve linked them for your convenience.

Azure Pipelines

Putting all of those previous tools together into one system, Azure Pipelines is my preferred open-source CI tool for a few key reasons. They support all three major operating systems: Windows, macOS, and Linux whereas other CIs seem to only strongly support a single system. It also is free for open-source projects and permits a higher number of parallel jobs.

Much like other CIs, Azure Pipelines moves towards the infrastructure as code (IaC) model where the pipeline configuration is specified in yaml form. Unlike other CI pipelines, you can break up those often complex and long pipelines into stages, jobs, and steps which can be split across files. The key file to know is the azure-pipelines.yml file which is the main file that the system looks for when running a build. One other key feature to note is that Azure Pipelines has a pretty dashboard which centralizes your tests and coverage tracking in a single source.

There are many awesome blogs that explain Azure Pipelines. If you want to understand this specific setup, take a look at the CI files to understand how they fit together.

If you want to look through a specific run of the build, take a look at this one which is configured to test on Windows, macOS, and Linux and across Python 3.6 and 3.7.

Documentation

Documentation is another key area that makes or breaks a good open-source project and more broadly speaking any programming project. While the value-add of a project might be stellar, without proper documentation a project’s adoption might be quite sluggish. It also lets us take advantage of all those fantastic docstrings we wrote along the way thanks to our linters.

sphinx, sphinx_rtd_theme

sphinx is the leading python documentation tool. It is a tool that, when mastered, can make some stunning documentation. Yet, it isn’t super difficult to get up and running. It uses ReStructured (rst) files to construct beautiful documentation websites. Using its rich extension system, you can enable napoleon, a Google style docstring parsing library, and autodoc, a library to inspect and automatically add your docstring to the HTML. Another neat feature is the ability to link between documentation by referencing objects for any other piece of documentation that is written in sphinx. Lastly, it works well with open-source services such as ReadTheDocs. If you would like to view your project as it would appear on readthedocs you need to install sphinx_rtd_theme and configure it in the conf.py as shown here. Feel free to check out the rest of the custom configurations in that file as well.

You can build the docs locally by running:

ReadTheDocs

ReadTheDocs is a great service that allows open-source projects to host documentation for free. It automatically links with GitHub to update the hosted documentation each time a new version of the code is pushed the main branch specified.

To set things up go to readthedocs:

And then:

Once again, I’ve already configured readthedocs to work with both poetry and my setup. If you are specifically interested in my setup, take a look at the source file. You can also take a look at the end result linked here.

Release

The last and most important piece in the Python tooling ecosystem surrounds project release. You want your users to have an easy way to get to your big project and these tools help make that process easy as one, two, three.

towncrier

As you get further along in the development process, you might realize that you need to create meaningful changelogs to help the user understand the differences between the versions of your code. towncrier lets you add those changes incrementally as you merge in pull requests. Come release time, this tool compiles and deletes newsfragments as the package likes to call them into a single changelog. The tool is configured with all of these settings in the pyproject.toml.

The first command does a draft run of towncrier compiling all the files in the specified newsfragments folder into a single draft changelog. The second command actually runs the tool and gives you the option of removing the squashed news fragments.

Poetry (release)

I mentioned that we would come back to poetry at the end and here we are. poetry is not only great for managing package dependencies but also for submitting a package to pypi. It’s really as simple as running poetry publish. If you pass your username and password the tool will add your package to the pypi repository. In addition to doing this, we would also recommend that you go to Github and mark a commit as the release for easy retrieval in the future.

Acknowledgements

Thanks to Diego Huang, Ji Chao Zhang, Jing Zhang, and David Poole for reading drafts of this blog post.

Relevant Links

Example Github: https://github.com/adithyabsk/simplecalc

Example Documentation: https://simplecalc.readthedocs.io/en/latest/

Install Steps: github gist (A compiled list of all install steps from the blog post).

TL;DR

Click on the names of the packages to jump to the relevant section.

Pyenv / pyenv-virtualenv

• What is it: A sane way to manage python versions and virtualenvs across packages.
• Why you should use it: It integrates well with tooling environments and enables testing across multiple python versions.

Poetry

• What is it: A python dependency manager, package installer, and package publisher all in one.
• Why you should use it: It has a multitude of convenience features, simplifies the project development lifecycle, uses semantic versioning, and is PEP518 compliant.

Flake8

• What is it: A python code linter that checks for code style errors and simple mistakes
• Why you should use it: It reduces the number of merge conflicts due to styling differences and makes your code more readable.

Plugins:

• flake8-docstrings: Checks your docstring style.
• darglint: Checks that your docstring input/output matches your code.

black

• What is it: An opinionated python code formatter that adheres to PEP8.
• Why you should use it: Fixes the majority of flake8 issues automatically.

isort

• What is it: A python import order sorter.
• Why you should use it: Prevents merge conflicts due to the addition of imports and makes imports more readable.

Plugins:

• seed-isort-config: Allows isort to identify external packages.

pre-commit

• What is it: A git pre-commit manager which automatically runs the above tools.
• Why you should use it: Prevents you from forgetting to run any of those tools before you commit.

pytest

• What is it: A package to simplify the process of writing python tests.
• Why you should use it: Provides an easy to use command-line interface and a plethora of plugins.

Plugins:

• pytest-cov: Integrates pytest with coverage.py.
• pytest-mock: Adds convenience mocking fixtures to pytest.
• xdoctest: Allows an easier integration with Google Style docstrings.

coverage

• What is it: A package to measure how well your unit tests validate your code.
• Why you should use it: Allows you to validate that your unit tests are valuable and track your codebase health.

tox

• What is it: A package to run your unit tests locally across multiple python versions.
• Why you should use it: Helps you in the debugging and testing process when faced with an error on only one python version.

Azure Pipelines

• What is it: A free continuous integration (CI) pipeline system that tests across all OSes and python versions.
• Why you should use it: Currently seems to be the leader in platform and plugin support for the Python ecosystem.

sphinx

• What is it: A package to help you write documentation for your code.
• Why you should use it: It’s the best documentation package out there with fantastic community support.

Plugins:

• sphinx_rtd_theme: Allows you to see how the docs would look on ReadTheDocs.

ReadTheDocs

• What is it: An open-source, free documentation hosting service.
• Why you should use it: It is the de-facto standard amongst the open-source community.

towncrier

• What is it: A package to allow you compile patch notes which are automatically added to a changelog.
• Why you should use it: Makes the process of writing release notes much easier.