Authoring a Modern Python Package

While relying on built-in package management facilities

Michael Orlov
13 min readAug 18, 2023
PyPI is the official third-party software repository for Python (Wikimedia Commons, GPL-2+)

I have recently experimented with creating a Python package while adhering to Python’s modern development and packaging principles. It turns out that writing, testing and maintaining cross-platform code is easy with the recent PEP specs and up-to-date core Python packages. All project settings can be maintained in a single pyproject.toml file (no setup.py).

I summarize below how these principles were applied to my open-source project curldl, including integration into GitHub CI/CD workflows which test and publish the package and its documentation on PyPI and Read the Docs. Project creation templates weren’t used, as everything is written from scratch. Advanced package management tools like Poetry proved to be unnecessary, as recent Setuptools provide all the required functionality.

Note also that the main run-time dependency is the native-code PycURL package with builds that are not trivially available on all platforms, making automated multi-platform testing more challenging than usual.

If you would like to upgrade your package to modern Python practices, while understanding how all the workflow components interact, or create a new package from scratch, read on.

Project Structure

Fashionable badges that show selected project features, with and without reason

These are the main files and directories of curldl, a package for safely and reliably downloading files via libcurl:

  • src/curldl — the package being developed; also contains the package CLI entry point __main__.py (for running as python3 -m curldl)
  • tests/* — the test packages, e.g. test/functional and test/unit
  • build — all build, tests and generated documentation outputs; not committed to VCS
  • docs — documentation and changelog settings; also contains generated RST files
  • README.md and LICENSE.md — self-explanatory; CHANGELOG.md is maintained automatically in docs
  • pyproject.toml — all project settings, including testing, linter, static typing, code styling and much more
  • .github/workflows — GitHub-specific CI/CD workflows, can be translated to another platform such as GitLab
  • venv.sh — virtual environment initialization and maintenance, very handy in CI/CD workflows
  • misc/scripts — helper scripts for running code and documentation analysis/generation tools with proper arguments
  • assets — miscellaneous resources such as PycURL Windows builds, maintained as a submodule (yuck, but GitHub limits LFS bandwidth)
Top-level view of the repository in GitHub

Just to get a sense of how the package can be used:

import curldl, os
dl = curldl.Curldl(basedir="downloads", progress=True)
# resumes from a partial downloads/linux-0.01.tar.gz.part file if present
dl.get("https://kernel.org/pub/linux/kernel/Historic/linux-0.01.tar.gz",
"linux-0.01.tar.gz", size=73091,
digests={"sha1": "566b6fb6365e25f47b972efa1506932b87d3ca7d"})
assert os.path.exists("downloads/linux-0.01.tar.gz")

Or from the command line:

# infers download filename from the URL
# supports non-HTTP(S) protocols, including download resume (if supported by protocol)
curldl -b downloads -s 73091 -a sha1 -d 566b6fb6365e25f47b972efa1506932b87d3ca7d \
-p -l debug ftp://ftp.hosteurope.de/mirror/ftp.kernel.org/pub/linux/kernel/Historic/linux-0.01.tar.gz

Virtual Environment

Running ./venv.sh upgrade-venv to upgrade the virtualenv environment in venv directory

The venv.sh script is responsible for setting up and using the virtualenv environment in venv directory. This script uses the Python installed in the system, supporting Linux/Windows platforms and CPython/PyPy interpreters (it should in principle work on macOS as well, but tests use Miniconda on that platform).

Note that I opted not to use the popular tox testing automation framework for multi-platform support, due in part to its poor integration with pyproject.toml project settings, but also due to a more optimal separation of responsibilities between CI/CD VMs (setting up the software environment) and project helper scripts (using the currently available interpreter).

C/CD workflow directly sets up Miniconda with Python 3.11 in macOS VM, without using virtualenv

The venv.sh helper script allows one to:

  • Install/upgrade the virtual environment packages, together with an in-place editable install of the package under development;
  • Downgrade the virtual environment packages to a minimal set of package dependencies, to be verified when smoke-testing the package;
  • Invoke any command in a virtual environment under hardened developmental Python settings, without entering the virtual environment beforehand, which is very useful in CI/CD workflow jobs.
CI/CD workflow installs a Python 3.11-based virtualenv environment in Windows VM

On Windows, venv.sh (run via a Bourne shell executable) installs the shipped PycURL builds from the assets submodule:

${pip} install --use-pep517 "${assets_dir}/pycurl-${pycurl_win32_build_version}-cp${python_short_version}-cp${python_short_version}-${pycurl_build_suffix}${python_bits}.whl"

In addition to venv.sh, misc/scripts directory contains wrapper scripts for direct invocation of all tools used by the project, including those typically invoked via pytest. This is useful both for command-line user interface and for CI/CD integration.

Usage examples:

# install virtual environment and run tests
./venv.sh install-venv
./venv.sh pytest

# reformat code, e.g. if Black that is run via pytest detects formatting issues
./venv.sh misc/scripts/run-black.sh

. venv/bin/activate # set up the shell for virtual environment
misc/scripts/run-sphinx.sh # generate documentation (used in CI/CD workflows)
curldl --version # entry point wrappers directory is in PATH

References:

Unified Project Settings

pyproject.toml configuration excerpt related to package and interpreter dependencies

The pyproject.toml file contains all project settings (but not documentation generation settings). The main sections are:

  • Project name, license and readme files, classifiers, URLs etc., to be imported by PyPI
  • VCS-based dynamic project version maintenance via setuptools-scm
  • Python interpreter and package dependencies, including extras
  • Command-line entry point for creation of wrapper scripts during installation (in addition to package-level __main__.py entry point)
  • Build system definitions — basically, setuptools and wheel, without complex package management frameworks
  • Declaration of type hints being shipped with the package
  • pytest default settings, with configuration section(s) for each tool that it invokes via a plugin: Pylint, Mypy, Coverage, Black and isort
  • Sections for tools that are not invoked via pytest: Bandit security checks and towncrier snippets-based Changelog maintenance

Note that while pytest passes the necessary directory arguments to the tools above via its plugins, the same paths can’t be configured for the tools in the project settings file. That’s one of the reasons for wrapper scripts in misc/scripts, should we need to run a tool as a standalone — e.g., running Black or isort when the respective plugin suggests modifying the code. The pytest plugins are configured to never modify code themselves and to only detect required formatting changes, due to their functionality in CI/CD workflows.

[tool.pytest.ini_options]
addopts = "--maxfail=10 --mypy --pylint --black --isort --cov --cov-report=term --cov-report=html --cov-report=xml"
testpaths = ["tests", "src"] # "src" required for pylint and mypy (whereas cov considers imported files)
cache_dir = "build/tests/pytest_cache"

References:

Unit Testing and Code Coverage

Running pytest 7.4.0 on Ubuntu 22.04 with Python 3.10.12 and venv setup

Proper unit testing is critical for producing a stable package that functions properly on multiple platforms, and pytest is indispensable for this purpose. The main advantage of pytest in my opinion is its excellent support for combinatorial parametrization of tests, which is used extensively in testing code. Plugins implementing various fixtures and wrapping execution of standalone tools like code coverage and static code analysis are also extremely useful.

HTML coverage report that is automatically created during pytest run

Automatic code coverage measurement using coverage.py deserves a separate mention here. curldl has been the first project where I implemented full code coverage via code tests, and I must say that it gives a very high level of confidence in the code and being able to freely modify it. Thus, I enabled a 100% coverage requirement for both production and test code, including branches coverage. It is achievable when developing a project from scratch, so your targets may differ — especially when porting legacy code — but good code coverage is really worth it in the long run.

Per-file HTML coverage report showing test contexts that covered a given line of code

An important technical note is that testing code shouldn’t reside in the same packages as the production code (something that is possible in e.g. Java), since it confuses the interpreter when trying to import modules from regular code or other testing packages. Therefore the testing packages are structured simply as tests/unit, tests/functional, etc. Here is a typical unit test with parametrizations and fixtures:

@pytest.mark.parametrize("size", [100, 5000])
@pytest.mark.parametrize("specify_size", [False, True])
@pytest.mark.parametrize("always_keep_part_size", [0, 50, 2499, 2501, 8000])
def test_partial_download_keep(
tmp_path: pathlib.Path,
httpserver: HTTPServer,
caplog: LogCaptureFixture,
size: int,
specify_size: bool,
always_keep_part_size: int,
) -> None:
"""Verify policy of keeping partial download in absence of verification data"""
caplog.set_level(logging.DEBUG)
with open(tmp_path / "file.txt.part", "wb") as part_file:
part_file.write(b"x" * (size // 2))

httpserver.expect_oneshot_request("/file.txt").respond_with_handler(
make_range_response_handler("/file.txt", b"y" * size)
)

dl = curldl.Curldl(
basedir=tmp_path,
verbose=True,
retry_attempts=0,
min_part_bytes=0,
always_keep_part_bytes=always_keep_part_size,
)
dl.get(
httpserver.url_for("/file.txt"),
"file.txt",
size=(size if specify_size else None),
)
httpserver.check()

assert not (tmp_path / "file.txt.part").exists()
assert read_file_content(tmp_path / "file.txt") == (
(b"x" * (size // 2) + b"y" * (size // 2))
if specify_size or always_keep_part_size <= size // 2
else b"y" * size
)

References:

Static Code Analysis and Formatting

Curldl class constructor in PyCharm, featuring mandatory type hints and docstrings, formatted with Black

Python is a great language that is often underestimated by functional language elitists. Case in point: SICP material can be and is taught using Python (in fact, I used to do exactly that). One issue with Python is however that it has no mandatory static typing — only optional type hints that are ignored by the interpreter. Therefore type hints only make sense when regularly enforced project-wise with a static type checker like mypy, which is exactly why mypy plugin is enabled in pytest, with strict settings.

Similarly, pylint plugin is enabled for automatically verifying code conformance with Python code style guides. There are opinions that pylint, while being thorough, is difficult to configure without producing many false positives, but I found no such issues when using it for a new project. This may be different when applying pylint to existing projects.

[tool.pylint.main]
py-version = "3.8"
ignore-paths = ["src/curldl/_version.py"]
extension-pkg-allow-list = ["pycurl"]

[tool.pylint.format]
expected-line-ending-format = "LF"
max-line-length = 88

[tool.pylint.basic]
good-names-rgxs = "^(dl)$"

[tool.pylint.design]
min-public-methods = 1
max-args = 10
max-attributes = 15
max-locals = 20

[tool.pylint.miscellaneous]
notes = ["FIXME", "XXX"]


[tool.mypy]
warn_unused_configs = true
cache_dir = "build/tests/mypy_cache"

allow_redefinition = true
warn_unreachable = true

disallow_untyped_defs = true
disallow_incomplete_defs = true
disallow_untyped_decorators = true
warn_incomplete_stub = true

disallow_any_explicit = true
disallow_any_generics = true
disallow_subclassing_any = true
warn_return_any = true

The two code formatting plugins, Black and isort, are excellent for taking over code formatting, enforcing a one-to-one correspondence of code text to its AST. I found that while you do lose some freedom of expressing yourself in line breaks and quote styles, you also gain the ability to forget about code formatting issues when reviewing someone else’s commits for integration. pytest integration only checks for formatting violations (intentionally, so that it can fail), and actual reformatting is done by running the corresponding scripts in misc/scripts.

Oh no! A false positive type hint issue detected by PyCharm. Fortunately, mypy is much more thorough.

It should be noted that when using PyCharm (by far the greatest Python IDE), the code analysis tools above are nearly transparent, including Black after getting used to its line breaks style and setting hard wrap to 88 characters. PyCharm might ignore missing type hints, but mypy will ensure that they are eventually added. It is, however, necessary to add external type annotation packages to test dependencies, if the package does not provide type annotations (e.g., types-tqdm).

In addition to the above, Bandit is used to scan the code for potential security issues. There is apparently no good pytest plugin for Bandit, and I didn’t feel like writing one myself because honestly I wasn’t impressed by this tool, so it is run from CI/CD workflows via a wrapper script.

CI/CD workflow in Windows VM with 32-bit Python 3.8 scanning for security issues with Bandit

References:

Generating Documentation

API Reference part of online package documentation

Automatically generating package documentation was a tough one, and the only part of the build cycle that is not configured via the pyproject.toml file. It was non-trivial for the following reasons:

  • Sphinx documentation generator feels like a mastodon from LaTeX and Texinfo era (big surprise that it was first released in 2008), and I didn’t want to rely on boilerplate Sphinx configuration files;
  • I prefer Markdown documentation markup to reStructuredText, however ReST snippets and generated files are still necessary;
  • Read the Docs integration with GitHub is quite inflexible (e.g., .readthedocs.yaml must be located at project’s root), and testing essentially consists of triggering the webhook and waiting.

Thus, the optimal way to achieve proper documentation publishing process is to have Sphinx locally generate the same HTML documentation tree that will be built on Read the Docs side, which is precisely what the Sphinx-running wrapper script does. Sphinx configuration resides in docs/conf.py, which also contains API reference ReST files generation from docstrings by sphinx-apidoc:

subprocess.run(
f"sphinx-apidoc -feM -o api ../src/curldl".split(), # nosec
check=True,
text=True,
encoding="ascii",
)
CI/CD workflow creating a downloadable documentation artifact

Sphinx is run from CI/CD workflows to generate downloadable artifacts, since Read the Docs builds the documentation independently after being triggered from a GitHub webhook. Note that a failure at this stage will intentionally fail the workflow due to bad docstrings or documentation files. The failure can be then investigated by running run-sphinx.sh locally or by reading the CI/CD output on GitHub.

References:

Matrix Testing via CI/CD Workflows

GitHub CI/CD workflow progress

Recall that at project level the tests presume a single Python instance. Setting up multiple platforms and Python implementations is achieved via GitHub actions that perform the setup at VM level. It’s a much more generic way of establishing varying testing environments than, e.g., those supported by tox. The workflows are configured in ci.yml file that must reside under .github/workflows. Note however that there is nothing GitHub-specific in the approach itself, and the same can be realized via e.g. GitLab or Jenkins.

GitHub’s way to convey unlimited actions time and storage for public repos (does not cover LFS transfers!)

The following configuration aspects are mix-matched in order to cover varying platform aspects while avoiding consumption of infinite resources, as gratuitously provided by GitHub to open-source projects:

  • Operating system: Linux, Windows, macOS (all 64-bit x86) — latest versions of GitHub-hosted runners; using non-latest versions of e.g. Ubuntu proved to be problematic and bringing little extra value
  • Python build: 64-bit x86, 32-bit x86
  • Python variant: CPython, PyPy
  • Python environment: venv, platform (using as many platform packages as possible, installing the rest without venv, including curldl wheel), Miniconda (see misc/conda/test-environment.yml)
  • Test environment: all latest packages, minimal required package versions, only sanity checks with just the basic required packages

Code coverage reports (one per OS), package build and documentation build are uploaded as artifacts, but care is taken to upload coverage even in case of failure, e.g.:

- name: Archive code coverage report
if: ((success() && matrix.python-version == '3.x' && matrix.platform == 'ubuntu') || steps.test.conclusion == 'failure') && !startsWith(matrix.python-version, 'pypy')
uses: actions/upload-artifact@v3
with:
name: coverage-report-${{ github.job }}-${{ matrix.python-version }}-${{ matrix.platform }}-${{ matrix.architecture }}
path: build/tests/coverage/
Downloadable CI/CD workflow artifacts

Coverage reports are also uploaded to Codecov (free for open-source projects), because what could we do without a badge leading to pretty graphs? Not too informative with 100% coverage, though.

Codecov Report as shown in GitHub’s pull request conversation

All workflows are triggered on pushes to the major branches and version tags, and also scheduled nightly.

GitHub CI/CD workflow triggered by a pull request to develop branch.

References:

Package Publishing via CI/CD Workflows

Excerpt from curldl package summary on PyPI with settings and description pulled from pyproject.toml

The ultimate goal of the CI/CD workflow is to publish the package to PyPI. Required preliminaries for publishing the package are building and thoroughly testing it as described previously, and these steps are performed regardless of whether the event that triggered the workflow activates the publishing step. Repository events that trigger publishing are version tag (v*) pushes.

CI/CD workflow builds and tests a Python package wheel, regardless of workflow trigger

To build the package’s source (sdist) and binary (wheel) distributions, the simple PEP 517 build frontend is completely sufficient; the result is then locally verified by Twine and tested with pytest. The resulting build artifact is later used by “platform” Python environment job during matrix testing and by the PyPI-publishing job, if activated with a version tag push.

Publishing to Test PyPI and to its operational counterpart via the CI/CD workflow is then straightforward via GitHub action Twine wrappers. However, it’s important to first verify publishing to Test PyPI from command line (via Twine), and ensure that everything works as intended, including PyPI pulling the correct project details from pyproject.toml settings.

- name: Publish distribution to Test PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.TEST_PYPI_CURLDL_TOKEN }}
repository-url: https://test.pypi.org/legacy/

- name: Publish distribution to PyPI if non-dev tag
if: (!contains(github.ref, 'dev'))
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.PYPI_CURLDL_TOKEN }}

The package publishing workflow also pushes a release to GitHub via the corresponding GitHub action, but in my opinion that’s entirely optional for a Python package that’s released to PyPI.

References:

--

--