Tox in Action: Testing Your Code Across Multiple Version Releases
Have you ever encountered a situation in the Python application lifecycle where you needed to test your code across various releases of a package, be it Python itself or any other?
If your answer is yes, then tox
is the solution you’ve been searching for.
Described in their own words:
tox
is an environment orchestrator.tox
aims to automate and standardize testing in Python. It is part of a larger vision of easing the packaging, testing and release process of Python software (alongside pytest and devpi).
Getting started with TOX:
To start using tox
we need to:
- Install
tox
- Configure
tox
usingtox.ini
configuration file - Make a
setup.py
orpyproject.toml
file.
Tox Installation:
$ pip install "tox<4"
Tox Configuration:
tox
needs a configuration file where you define what tools you need to run and how to provision a test environment for these. The canonical file for this is the tox.ini
file
E.g:
A minimal tox.ini
can look like this
[tox]
requires =
tox<4
env_list = py{38,39}
skipsdist = True
[testenv]
deps =
pytest
commands =
pytest {posargs: .}
The configuration is split into two type of configuration: core settings are hosted under a core tox
section while per run environment settings hosted under testenv
and testenv:<env_name>
sections.
We start by putting a constraint that this config requires tox to be on a version less than 4. Then we define the environments(env_list
) to run our tox commands against. In this case we have python3.8 and python3.9. We will come to the skipdist
option later in the next section.
Full list of supported configurations can be seen here in the official docs.
In our environment settings([testenv]
), we define the dependencies to install under the deps
section. The dependencies can be listed explicitly here as well as we can provide a requirement file. Finally we define the commands
to run.
Package Configuration(Make a setup.py
or pyproject.toml
file):
This is where the skipdist
option comes into play. If you are working in a project which is a python package or library then you would have either of setup.py
or pyproject.toml
file located in your project. If it is not the case then you can specify skipdist
as True
If you wish to create a `setup.py` or pyproject.toml file then you can take guidance from the below mentioned links:
How to run:
Now that we are done configuring tox, we can actually see it in action.
To run tox
with all the environments that we have specified simply go to the terminal and at your project root where tox.ini
is located run:
$ tox
To test a single environment we can pass the -e
flag with the name of the environment, i.e. for executing just python 3.9 run:
$ tox -e py39
If we ever change the dependencies in the environment after initial run we have to recreate the environment in order to install the deps again. To do this run:
$ tox --recreate
Advanced usage:
Testing with multiple versions of multiple packages:
Say we have multiple versions of a package which we want to test against multiple versions of other packages. Take for example Celery, then in a production grade application we would like to test multiple versions of Celery against multiple versions of Django and Python at the same time.
We can even achieve this using tox
[tox]
requires =
tox<4
envlist = py{38,39}-django{32,42}-celery{44,50}-drf{latest}
skipsdist = True
[testenv]
deps =
django32: Django>=3.2,<4.0
django42: Django>=4.2,<4.3
drflatest: djangorestframework
celery44: celery>=4.4,<4.5
celery50: celery>=5.0,<5.1
pytest
commands =
pytest {posargs: .}
Here we defined envlist that contains multiple versions of python, django and celery and latest version of DRF. This is termed as generative environments.
The environments generated for this tox
configuration are as follows:
py38-django32-celery44-drflatest
py38-django32-celery50-drflatest
py38-django42-celery44-drflatest
py38-django42-celery50-drflatest
py39-django32-celery44-drflatest
py39-django32-celery50-drflatest
py39-django42-celery44-drflatest
py39-django42-celery50-drflatest
We can see the complete list of generated environments using the command: tox -l
Allow other system commands within tox:
By default commands outside of tox
environment cannot be used inside tox. For using such commands we need to explicitly allow them in the configuration.
Let say I have a testing environment named docs and in that I want to use the linux commands make
and rm
then I would allow them in tox
with below configuration:
[testenv:docs]
allowlist_externals =
make
rm
After putting the commands in allowlist
now I can use them in the commands section of the environment.
[testenv:docs]
allowlist_externals =
make
rm
commands =
rm -f docs/modules.rst
make -C docs clean
Set environment variables within tox:
To set some environment variables within tox environment we need to also define them in tox configuration. We can do something like the following:
[testenv]
setenv =
DJANGO_SETTINGS_MODULE = test_settings
PYTHONPATH = {toxinidir}
Final Thoughts:
Last thing I would love to mention here is the tox CLI
. It provides a whole lot of options to us for playing around. We can explore all those options by running tox -h
command in the terminal.
All that said, tox
is a versatile tool that has a wide range of capabilities not just limited to those described in this article. We can even extend tox
by writing custom plugins for which we can find helpful pointers in tox’s
own documentation.