Unit Testing in Python — tox and nox

Develop Fast, Test Thoroughly

Martin Thoma
Jul 23 · 4 min read
Image for post
Image derived by Martin Thoma from Andrea Caprotti (nox project)

When I started developing Python packages, there was one mistake I did quite often: I forgot to add all dependencies. Additionally, I only tested on my machine for a single Python version.

After reading this article, you will know how to locally and automatically test multiple Python versions in isolated environments. This is a preparation for Continuous Integration tools like Travis. I assume you already know the basics of unit testing in Python and how to package your code.

pyenv

pyenv is a tool which lets you easily install and switch Python environments on your system. Take a look at the official installation instructions; it’s not a Python package but hooks directly into your shell.

Once it is installed, you can get a list of all available Python environments. In July 2020, there were 427 different versions! I’ve shortened them here to show you the ones I think are interesting:

$ pyenv install --list
Available versions:
[...]
2.7.18
[...]
3.6.11
[...]
3.7.8
[...]
3.8.4
3.9.0b4
3.9-dev
3.10-dev
[...]
pypy-c-jit-latest
pypy-dev
[...]
pypy-5.7.1
[...]
pypy3.6-7.3.1
[...]

Install a Python version:

$ pyenv install 3.8.4
Downloading Python-3.8.4.tar.xz...
-> https://www.python.org/ftp/python/3.8.4/Python-3.8.4.tar.xz
Installing Python-3.8.4...
Installed Python-3.8.4 to /home/moose/.pyenv/versions/3.8.4

And use it:

$ pyenv local 3.8.4$ python --version
Python 3.8.4
$ pip --version
pip 20.1.1 from /home/moose/.pyenv/versions/3.8.4/lib/python3.8/site-packages/pip (python 3.8)

Virtual environment basics

A virtual environment encapsulates the installed packages. Different virtual environments still share the same operating system, the same installed c libraries and executables. The only difference is which packages are available.

You can create a new virtual environment called venv-tutorial like this:

python -m venv venv-tutorial

It creates a folder which contains all installed packages and few other things. To use it, you need to activate it:

source venv-tutorial/bin/activate

This will add the prefix (venv-tutorial) in front of your prompt. It will make sure that later calls to python and pip use this environment. If you want to stop it again, type deactivate in the shell. If you delete this folder, the virtual environment is gone.

Testing multiple Python versions without tox

When you claim that your project supports Python 2.7 and 3.5 to 3.8, then you better test those versions. In order to make sure that you install the packages properly, you should create a virtual environment. You might end up with creating a shell script which creates those virtual environments, starts the tests and deletes the virtual environments again.

How to use tox

tox uses a tox.ini file which is in the package root directory. So your project structure might look like this:

your-awesome-project/            # The git repository
├── README.md
├── setup.py # Dependencies / Package Meta data
├── your_awesome_package/ # Code of the package
│ ├── a_module.py
│ └── another_module.py
├── tests/ # Unit tests
│ ├── test_a_module.py
│ └── test_another_module.py
└── tox.ini # Why you're reading this article

The tox.ini file to run the tests in an isolated Python environment for Python 3.6, Python 3.7 and Python 3.8 looks like this:

Note that you need to have the different Python versions already installed. You can do this with pyenv and make them available with the following command:

$ pyenv local 3.8.4 3.7.8 3.6.11

Run tox within the root folder — the same folder that contains your tox.ini file.

The next thing you might want to do is to break a couple of things out of the pytest run. For example, it’s not necessary to run the linter flake8 and blackin every single environment. Instead, you can define a linter environment which is run once:

And finally, you want to run the different environments in parallel for speed:

tox -p

If one of them fails, you get this type of output:

Image for post
Run tox in parallel, showing a linter issue. This output is way cleaner than if you had run flake8 and black via pytest. It might be a tiny bit faster, but that difference is not relevant. Screenshot taken by Martin Thoma

If you want to run just the linter, tox -e linter is your friend 🙂

You can also create a matrix of different combinations of Python environments and dependencies you install. Thea Flowers showed this at PyCon 2019 (video), but I never had the need to go down that rabbit hole.

Now… what is nox?

nox is a spin-off of tox. Instead of using a tox.ini configuration file, it uses a noxfile.py Python file. It’s pretty similar to tox, but more flexible as it uses Python code:

You can run a single session of nox with nox -s lint:

Image for post
Screenshot taken by Martin Thoma

Overall, not a huge difference. The output of nox is way nicer and for people who get started, I think the Python code written with nox is a bit simpler to read and understand than the tox.ini configuration file.

What’s next?

In this series, we already had:

In future articles, I will present:

  • Static Code Analysis: Linters, Type Checking, and Code Complexity

Let me know if you’re interested in other topics around testing with Python.

Python In Plain English

Go deeper with the language powering everything.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store