Understanding Best Practice Python Tooling by Comparing Popular Project Templates

A look at well-known and up-and-coming Python tools

Jonas Kemper
Feb 10 · 6 min read

Target audience: Intermediate Python programmers.

TL;DR: Use Flake8, pytest, and Sphinx in your current Python project. Also evaluate pre-commit, Black, and Pylint. For your next project, consider adding Poetry and Dependabot.


Overview

I always find it difficult to have a balanced opinion about Python best-practices. The hype-driven tech world makes it difficult to filter signal from noise.

A newly-advertised tool often sounds great on paper, but is it actually making me a more effective engineer? Or is it just one more thing I need to look after, adding more complexity than value?

I had a vague idea of what I considered best-practice, but it was mostly based on anecdotal evidence and casual conversations. Then, in the last couple of weeks, I started to look at all the Python project templates (i.e. cookiecutters) I could find.

To me, it seemed extremely interesting to see which tools the creators of these templates deemed worthy of being part of their scaffolding for every new Python project.

I compared the 18 most popular (ranging from 76 to 6.3k GitHub stars) project templates with an emphasis on which tools they endorsed. The results can be found in this spreadsheet:

In the sections below, I’m highlighting my key take-aways.


The De-Facto Standards

The tools in this section have been included in significantly more than half of the templates, which means they have been adopted as defaults by large numbers of Python projects out in the wild.

Flake8

I’ve used Flake8 for quite a while, but I wasn’t aware of how dominating it is. I thought there was some competition, but the vast majority of project templates rely on it.

No wonder, the convenience of linting entire codebases in seconds is hard to argue against. To be even more cutting-edge, consider wemake-python-styleguide as a kind of “Flake8 on steroids” that even has the potential to make other tools (e.g. Pylint) obsolete.

Pytest and coverage.py

The vast majority of templates rely on pytest and thereby disincentivize the use of the standard library’s unittest. Shines brightest when paired with tox, which about half of the templates included in their setup.

Sphinx

For the majority of templates, Sphinx is the go-to tool for documentation generation. To my surprise, MkDocs only ranks as a distant second.

So what? If you’re not using Flake8, pytest, and Sphinx in your current Python project, consider adopting them.


Up and Coming

In this section, I collect what I have observed in the templates as trends. This means that while not yet being supported by a majority of project templates, most of the younger templates encourage the use of the following tools and conventions.

Pyproject.toml

Pyproject.toml: PEP 518 proposed this modernized way of specifying build requirements, and most younger templates have adopted it.

Poetry

While the Python ecosystem has been off to a rough start regarding a strong tool for dependency management, I’m cautiously optimistic that Poetry can become Python’s npm.

The younger (but still popular) project templates seem to agree. But do keep in mind that if you are working on some kind of library which you might want to distribute via PyPI, you would still opt for setuptools instead.

(Update: I received feedback after publishing that this does not seem to be an issue anymore.)

Also, be careful if you (or your dependencies) are relying on Conda. In that case, you will not want to use Poetry because, in its current form, it ties you to pip and virtualenv.

Dependabot

Dependabot frequently checks for outdated dependencies and tries to help you by automatically opening a PR.

I’ve personally seen this added to a lot more projects recently, and it seems to be a great addition with a lot of upsides for potentially mitigating security threats by pushing engineers to keep dependencies up-to-date.

So what? Keep your eyes open for Poetry and Dependabot. Consider adopting them for your next project.


Personal Recommendations

The analysis of the project templates gave me a bit of an ambivalent answer to the tools listed below. Nevertheless, I would like to use this section to personally endorse them, because they have in the past provided great value for me.

Pre-commit

Pre-commit: Even if you have a lot of discipline, don’t waste it on something like checking for linter errors one more time before Git committing/pushing. Instead, save your discipline for good TDD and mob-programming practices.

Pylint

Being criticized for running too slow and being too bureaucratic to administrate, Pylint has made (and still is making) me a better engineer.

It gives me specific pointers for where to improve my code and how to better comply with convention. That, by itself, is a lot of value coming from a free tool, which makes it worth the hassle for me.

Black

“Uncompromisingly” eliminating all ambiguity about where to put whitespace in Python code, Black saves our teams from countless low-value discussions and low-value diffs caused by diverging editor configs.

To me, it mitigates one of my personal Python negatives (i.e. significant whitespace) to be a non-issue. Moreover, in 2019, Black has moved under the Python Software Foundation umbrella, which is a strong endorsement.

So what? If you’re not using pre-commit, Black, and Pylint, evaluate if they could add value for your team.


Meta and Bottom-Line

Twelve out of the 18 templates I analyzed have been built with cookiecutter. Some of the non-cookiecutter templates have some interesting features.

But given cookiecutter’s status as the go-to templating framework, you should have a good reason and make a conscious decision if you want to use one of the non-cookiecutter templates.

Choose a template that matches your vision most-closely. If you will require the same sort of customization on top of a given template frequently, fork and customize it by taking inspiration from more than one template in my list.

And if you’re feeling adventurous, create your own template from scratch. Cookiecutter is a great part of the Python ecosystem and straightforward Jinja2 templating makes creating your own a breeze.


Bonus: Template Recommendations

Django

Next to the most popular Django templates, consider also wemake-django-template. It gives a very well thought-through impression.

Data science

It most scenarios, you would want to use Cookiecutter Data Science, however, you should also check out Kedro.

It extends Cookiecutter Data Science to include a way for you to build standardized data pipelines and has load/save support for data and models. These features might make it a perfect fit for your next project.

I would also like to leave a hat tip to shablona for its super-useful documentation, which might contain some value for you, even if you end up using something else.

General-purpose

This depends a bit on whether you’re building a library or an application, but next to the most popular templates, I would personally give Jace’s Python Template a good look.

It is one of the less-popular templates, but I like that it comes out-of-the-box with Poetry, isort, Black, pylint, and mypy.

PyScaffold is the most-popular non-cookiecutter template and has many extensions (e.g. Django, Data science). And it pulls version numbers from Git via setuptools-scm, which is great. Also, it is among the few templates that support Conda.

Two of the templates come with GitHub Actions included:

  1. The first is Python Best Practices Cookiecutter which also comes with most of my favorite tools included.
  2. The other one is Blueprint/Boilerplate For Python Projects which I think is interesting because finding common security issues with Bandit sounds promising — and bundling all tool configuration in a single setup.cfg is brilliant.

Lastly, check out wemake-python-package — especially, but not exclusively, if you like their Django template or if you're interested in using they're super-charged wemake-python-styleguide instead of vanilla Flake8.


Conclusion

Update — I received some feedback for this blog post from Guido, and you might also find it interesting:


Contributors

Contributors/proof-readers: Martin Heinz, Matthias Misiewicz, Friedemann Altrock, Andrej Marsic, Takahiko Ito, Brendan Maginnis, Radovan Bast, Ariel Rokem, Nikita Sobolev, Florian Wilhelm, Yetunde Dada

Better Programming

Advice for programmers.

Jonas Kemper

Written by

Software engineer @McKinseyDigital | often found coding Python | Berlin | https://twitter.com/jonasrk | https://github.com/jonasrk

Better Programming

Advice for programmers.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade