Tox in Action: Testing Your Code Across Multiple Version Releases

Have you ever encountered a situation in the Python application lifecycle where you needed to test your code across various releases of a package, be it Python itself or any other?

If your answer is yes, then tox is the solution you’ve been searching for.

Described in their own words:

toxis an environment orchestrator. tox aims to automate and standardize testing in Python. It is part of a larger vision of easing the packaging, testing and release process of Python software (alongside pytest and devpi).

Getting started with TOX:

To start using tox we need to:

1. Install tox
2. Configure tox using tox.ini configuration file
3. Make a setup.py or pyproject.toml file.

Tox Installation:

$ pip install "tox<4"

Tox Configuration:

tox needs a configuration file where you define what tools you need to run and how to provision a test environment for these. The canonical file for this is the tox.ini file

E.g:

A minimal tox.ini can look like this

[tox]
requires =
    tox<4
env_list = py{38,39}
skipsdist = True
[testenv]
deps =
  pytest
commands =
  pytest {posargs: .}

The configuration is split into two type of configuration: core settings are hosted under a core tox section while per run environment settings hosted under testenv and testenv:<env_name> sections.

We start by putting a constraint that this config requires tox to be on a version less than 4. Then we define the environments(env_list) to run our tox commands against. In this case we have python3.8 and python3.9. We will come to the skipdist option later in the next section.

Full list of supported configurations can be seen here in the official docs.

In our environment settings([testenv]), we define the dependencies to install under the deps section. The dependencies can be listed explicitly here as well as we can provide a requirement file. Finally we define the commands to run.

Package Configuration(Make a setup.py or pyproject.toml file):

This is where the skipdist option comes into play. If you are working in a project which is a python package or library then you would have either of setup.py or pyproject.toml file located in your project. If it is not the case then you can specify skipdist as True If you wish to create a `setup.py` or pyproject.toml file then you can take guidance from the below mentioned links:

Packaging and distributing projects - Python Packaging User Guide

Writing your pyproject.toml - Python Packaging User Guide

How to run:

Now that we are done configuring tox, we can actually see it in action.

To run tox with all the environments that we have specified simply go to the terminal and at your project root where tox.ini is located run:

$ tox 

To test a single environment we can pass the -e flag with the name of the environment, i.e. for executing just python 3.9 run:

$ tox -e py39

If we ever change the dependencies in the environment after initial run we have to recreate the environment in order to install the deps again. To do this run:

$ tox --recreate

Advanced usage:

Testing with multiple versions of multiple packages:

Say we have multiple versions of a package which we want to test against multiple versions of other packages. Take for example Celery, then in a production grade application we would like to test multiple versions of Celery against multiple versions of Django and Python at the same time.
We can even achieve this using tox

[tox]
requires =
    tox<4
envlist = py{38,39}-django{32,42}-celery{44,50}-drf{latest}
skipsdist = True
[testenv]
deps =
  django32: Django>=3.2,<4.0
  django42: Django>=4.2,<4.3
  drflatest: djangorestframework
  celery44: celery>=4.4,<4.5
  celery50: celery>=5.0,<5.1
  pytest
commands =
  pytest {posargs: .}

Here we defined envlist that contains multiple versions of python, django and celery and latest version of DRF. This is termed as generative environments.

The environments generated for this tox configuration are as follows:

py38-django32-celery44-drflatest
py38-django32-celery50-drflatest
py38-django42-celery44-drflatest
py38-django42-celery50-drflatest
py39-django32-celery44-drflatest
py39-django32-celery50-drflatest
py39-django42-celery44-drflatest
py39-django42-celery50-drflatest

We can see the complete list of generated environments using the command: tox -l

Allow other system commands within tox:

By default commands outside of tox environment cannot be used inside tox. For using such commands we need to explicitly allow them in the configuration.

Let say I have a testing environment named docs and in that I want to use the linux commands make and rm then I would allow them in tox with below configuration:

[testenv:docs]
allowlist_externals = 
    make
    rm

After putting the commands in allowlist now I can use them in the commands section of the environment.

[testenv:docs]
allowlist_externals = 
    make
    rm
commands = 
    rm -f docs/modules.rst
    make -C docs clean

Set environment variables within tox:

To set some environment variables within tox environment we need to also define them in tox configuration. We can do something like the following:

[testenv]
setenv = 
    DJANGO_SETTINGS_MODULE = test_settings
    PYTHONPATH = {toxinidir}

Final Thoughts:

Last thing I would love to mention here is the tox CLI. It provides a whole lot of options to us for playing around. We can explore all those options by running tox -h command in the terminal.

All that said, tox is a versatile tool that has a wide range of capabilities not just limited to those described in this article. We can even extend tox by writing custom plugins for which we can find helpful pointers in tox’s own documentation.