Setting Your Python Project Up for Success in 2024

Python continues to evolve, and with each passing year, new tools and best practices emerge for structuring Python projects. In this guide, we will explore the key components of a high-quality Python project going into 2024. We will first cover the general project structure and the essential files and configurations you should have. We then go on to cover different aspects of a Python project that make it a high-quality project. Following this guide, you will end up with a project that is up-to-date with the most recent developments in the Python ecosystem.

If you have just started using Python: Don’t worry if all of this seems overwhelming. Just keep coding and eventually you will probably bump into structural problems that this guide can help you with.

If you are a Python veteran: I know that there are countless other ways to achieve what I am describing in this guide. Feel free to leave feedback and maybe you even find some inspiration.

Ready? Let’s go!

Project Structure

A well-organized project structure is crucial for maintaining a clean and understandable codebase. The “src layout” is a popular choice for Python projects because it prevents you from shooting yourself in the foot with some of Python’s quirks. This structure separates your source code from other project assets, making it easier to manage your code. Here’s what the project structure looks like:

my-project/
├── src/
│ └── my_project/
│   ├── __init__.py
│   └── example.py
│
├── tests/
│ └── ...
│
├── docs/
│ └── ...
│
├── pyproject.toml
├── tox.ini
├── LICENSE.txt
└── README.rst

Let’s break down each of these components:

• src/: This directory is where you store your project’s source code. The structure inside src/ represents your package’s structure, allowing for clean and organized code.
• tests/: This directory contains tests for your code. Every high-quality Python project should contain tests that make sure that your code works as intended.
• docs/: This directory contains documentation for your Python project.
• pyproject.toml: This file contains almost all configuration required to work on your project. Many tools like linters, build backends, and test runners can be configured in a pyproject.toml file.
• tox.ini: tox is a tool for automating test environments, which can also be used to automate all kinds of processes in a reproducible manner. We will use tox as the driver for our pipeline. The tox.ini file contains the configuration for running tox environments.
• LICENSE.txt: Every project should include a license file that specifies the terms under which the code can be used, modified, and distributed.
• README.rst: This is what many people will see first when looking at your project. A well-written README provides essential information about your project, such as installation instructions, usage examples, and contributions guidelines.

Make the Project Installable

Making the project installable is the first step that distinguishes your project from just a collection of Python scripts. The easiest way to make your project installable is to build it as a package. Open the pyproject.toml file and add the following to it:

[build-system]
requires = ["setuptools>=68", "setuptools_scm[toml]>=8"]
build-backend = "setuptools.build_meta"

[project]
name = "my-project"
requires-python = ">=3.8"
dynamic = ["version"]
dependencies = [
    # Add runtime dependencies here
]

# Enables the usage of setuptools_scm
[tool.setuptools_scm]

This tells tools like pip that they should use setuptools as the build backend. Furthermore, it activates setuptools_scm to automatically retrieve the package version from your Git history. Setuptools reads the information in the [project] table to build a package from your project. This is only a minimal example and you should add more metadata later on if you want to publish your package.

You can now install your project with pip to work on it. We use a virtual environment for that. Run the following from your terminal:

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -e .

This creates a virtual environment, activates it, upgrades pip and then installs your project in editable mode. Running pip list should show your project as installed. You can now enter Python (run python from your terminal) and import your package with from my_project import example.

Setting Up a Pipeline

You could now start hacking away on whatever you are building but we want to build a high-quality project, so we get our pipeline up and running first. Trust me, you really want to do that now before you start writing any actual code, so you keep your project nice and tidy and don’t have to clean up a mess later.

We will use tox as the driver for the pipeline to

• lint the source code,
• run tests,
• build the project’s documentation,
• build a package for publication.

tox makes this process reproducible, so that other developers can check out the project and get up and running quickly with just a handful of commands.

Add the following to the tox.ini file:

[tox]
envlist =
    lint
    {py38,py39,py310,py311}-test
    combine-test-reports
isolated_build = True


[testenv:lint]
description = Run static checkers.
basepython = py38
extras = lint
commands =


[testenv:{py38,py39,py310,py311}-test]
description = Run doc tests and unit tests.
extras = test
commands =


[testenv:combine-test-reports]
description = Combine test and coverage data from multiple test runs.
depends = {py38,py39,py310,py311}-test
commands =


[testenv:docs]
description = Test and build the docs.
extras = docs
commands =


[testenv:build]
description = Build the package.
extras = build
commands =

This configuration tells tox that there are five different steps in our pipeline. One nifty feature of tox is that it can run pipelines with different Python versions. Linting should be performed against the minimum Python version that your project supports (in our case Python 3.8, which is the lowest officially supported Python version). Tests should be executed against all supported Python versions. The combine-test-reports environment takes care of merging the test results from tests for different Python versions, after they have been executed. Building the package and the docs does not depend on a specific Python version, so we just omit that information. We will fill the commands section later to make the pipeline actually do something.

Every section of our pipeline will depend on external packages, like Mypy, Pytest, or Sphinx. We add these dependencies as so-called extras. tox will use these extras to install the required dependencies for the different parts of the pipeline. Add the following to your pyproject.toml :

[project.optional-dependencies]
lint = [
]
test = [
]
doc = [
]
build = [
]

We’ll fill in the actual dependencies later. Sometimes you might want to run just a single tool from the pipeline, so we will use a little trick to make that very easy. Add the following to your pyproject.toml file under the [project.optional-dependencies] section:

dev = [
    "tox",
    "my-project[lint]",
    "my-project[test]",
    "my-project[doc]",
    "my-project[build]",
]

The “dev” extra includes all the other extras of your package. Getting ready to work on your project after checking it out now only requires running pip install -e .[dev]. This installs the project in editable mode and also installs tox and all extras of your project.

The workflow for working on your project thus becomes:

1. Check out the project’s repository
2. Create and activate a virtual environment
3. pip install -e .[dev]
4. tox run -e lint to lint your project.
5. tox run -f test to run tests for your project. Note the -f flag which will execute all tox environments that have a test factor in their name. This will run tests against all specified Python versions and then aggregate the test results.
6. tox run -e docs to build the project’s documentation.
7. tox run -e build to build a package from your project.

Pro tip: You can define which steps get run when simply executing tox run by adding them to the envlist in your tox.ini file.

Congratulations, you now have a project that does nothing and a pipeline that also does nothing ;) However, all structural elements to ensure your projects sustained success are in place and only need to be filled with life.

Linting / Static Code Analysis

Linting, also known as static code analysis, is a critical aspect of modern software development, especially in dynamic languages like Python. It involves analyzing source code to identify potential errors, stylistic issues, and problematic patterns before the program is run. Effective linting can significantly improve both the quality and maintainability of your codebase.

Linting can help you with:

1. Error prevention: Linting tools detect syntax errors, undeclared variables, potential bugs, and other common coding mistakes.
2. Code consistency: They enforce a consistent coding style across the entire code base, which is crucial for collaborative projects.
3. Improved readability: Consistent coding styles and best practices enhance the readability and understandability of the code.
4. Educational value: They help developers, especially those new to a language or a code base, learn best practices and common conventions.

The world of Python tooling is a bit of a beautiful mess, with no central authority and a rapidly evolving ecosystem. One of the more drastic recent trends is the emergence of Ruff as the single tool for all static code analysis needs. It stands on the shoulders of giants like isort, flake8, Pylint, pydocstyle, and many others but it reimplements their checks in Rust, making them execute extremely fast. In my opinion, Ruff is the only linting tool you’ll need in 2024 and beyond. In addition, its formatting capabilities are quickly converging with Black’s (the de-facto standard in the Python world) and will probably take over that space as well. The only thing remaining then is type checking and I am sure people will get to that soon. So we really live in exciting times where a pipeline that previously consisted of 5–10 different tools (with different CLIs and configurations) converges to 1–3 tools instead (and maybe just one in the future?).

Long story short, we will use two tools for our linting pipeline:

1. Ruff — The formatter and linter

Ruff provides a formatter that enforces a consistent and opinionated coding style. It automatically formats your code to adhere to the PEP 8 style guide. Using the Ruff formatter ensures that your code is consistently styled, making it more readable and maintainable.

Ruff also checks your code for potential errors, enforces coding standards, and identifies code smells. For example, it takes care of ordering your imports, checks your docstring styles, and keeps you from making dumb mistakes. It even gives you the option to automatically fix a lot of those issues for you.

2. Mypy — The type checker

Mypy helps you catch type-related errors early in the development process. By adding type hints to your code and running Mypy as part of your linting pipeline, you can improve code quality, enhance readability, and reduce the risk of type-related runtime errors.

We first add the dependencies to the lint extra in the pyproject.toml :

lint = [
    "mypy",
    "ruff",
]

We then fill the lint step of the pipeline we defined in tox.ini with life:

commands =
    # Check formatting
    ruff format . --check
    # Lint code and docstrings
    ruff check .
    # Check type hinting
    mypy .

We can specify Ruff’s and Mypy’s behavior in the pyproject.toml as follows. Feel free to adjust the settings to your needs.

[tool.ruff]
line-length = 120
src = ["src"]
extend-exclude = [
    "conf.py",
]
target-version = "py38"
select = ["ALL"]
ignore = [
    "COM812",   # Conflicts with the formatter
    "ISC001",   # Conflicts with the formatter
    "ANN101",   # "missing-type-self"
    "PT001",    # https://github.com/astral-sh/ruff/issues/8796#issuecomment-1825907715
    "PT004",    # https://github.com/astral-sh/ruff/issues/8796#issuecomment-1825907715
    "PT005",    # https://github.com/astral-sh/ruff/issues/8796#issuecomment-1825907715
    "PT023",    # https://github.com/astral-sh/ruff/issues/8796#issuecomment-1825907715
]

[tool.ruff.per-file-ignores]
"tests/**" = [
    "S101", # Use of `assert` detected
    "D103", # Missing docstring in public function
]
"**/__init__.py" = [
    "F401", # Imported but unused
    "F403", # Wildcard imports
]
"docs/**" = [
    "INP001",   # Requires __init__.py but docs folder is not a package.
]

[tool.ruff.lint.pyupgrade]
# Preserve types, even if a file imports `from __future__ import annotations`(https://github.com/astral-sh/ruff/issues/5434)
keep-runtime-typing = true

[tool.ruff.pydocstyle]
convention = "google"

[tool.mypy]
disallow_untyped_defs = true # Functions need to be annotated
warn_unused_ignores = true
exclude = [
    "my-project-\\d+", # Ignore temporary folder created by setuptools when building an sdist
    "venv.*/",
    "build/",
    "dist/",
]

Run ruff format . to format your code or use the VS Code extension to automatically format your code whenever you save your code.

Ruff and Mypy will give you loads of feedback on your code, without even executing it. This can sometimes be a bit overwhelming and it is important to keep in mind that these tools are there to help you, make you aware of actual errors, and keep you honest when writing smelly code. However, forcing every line of code to comply even with the most rigorous rules is sometimes not the best use of your time and you should make sparing use of a # type: ignore or # noqa comment to ignore a warning.

Testing

Writing tests for your code is important for several reasons:

1. Error detection: Tests help identify bugs and errors early in the development process, making them easier and less costly to fix.
2. Code quality: Tests encourage you to write more modular and maintainable code, as testable code often aligns with good software design principles.
3. Refactoring confidence: With a good test suite, you can refactor and improve your code with confidence, knowing that tests will catch any regressions.
4. Documentation: Tests serve as practical documentation for how your code is supposed to behave, aiding both current and future developers.
5. Development efficiency: Automated tests can significantly speed up the development process by reducing manual testing efforts.
6. Long-term reliability: In a long-term project, tests help ensure that new features or fixes don’t break existing functionality.

Let’s finally write and execute some code. Add a simple function to example.py :

"""Contains an example function."""


def my_function() -> int:
    """Always returns 42.

    Returns:
        Always 42.
    """
    return 42

Now add a corresponding file test_example.py in the tests folder:

"""This module contains example tests."""

from my_project.example import my_function


def test_my_function() -> None:
    expected_value = 42
    assert my_function() == expected_value

We can use pytest to execute our test and thus make sure that our code works as expected. To make sure that our code works for all the Python versions that we support, we will run pytest from within tox. We will also measure code coverage to make sure that we did not forget to write tests for some crucial code path. First, we need to add the dependencies to our test extra in pyproject.toml :

test = [
    "pytest==7.4.1",
    "pytest-cov==4.1.0",
    "coverage[toml]==7.3.1",
]

As soon as you start importing pytest in your test code, for example to write fixtures or to use pytest.raises, you will have to add pytest to your linting dependencies as well, so better do that right away.

Next, we fill our test-related steps in tox.ini with life:

[testenv:{py38,py39,py310,py311}-test]
description = Run doc tests and unit tests.
package = wheel
extras = test
setenv =
    PY_IGNORE_IMPORTMISMATCH=1 # https://github.com/pytest-dev/pytest/issues/2042
    COVERAGE_FILE = reports{/}.coverage.{envname}
commands =
    # Run tests and doctests from .py files
    pytest --junitxml=reports/pytest.xml.{envname} {posargs}


[testenv:combine-test-reports]
description = Combine test and coverage data from multiple test runs.
skip_install = true
setenv =
    COVERAGE_FILE = reports/.coverage
depends = {py38,py39,py310,py311}-test
deps =
    junitparser
    coverage[toml]
commands =
    junitparser merge --glob reports/pytest.xml.* reports/pytest.xml
    coverage combine --keep
    coverage html

We are not gonna go into the details of every line here. The tox documentation has a very good introduction on writing tox.ini files. Basically, when you execute tox run -f test, tox creates a virtual environment for each of the Python version you want to test against. It then executes pytest in each of the environments and writes the tests results and the coverage report to files in the reports folder. The combine-test-reports step then collects the test results and coverage reports and merges them together into a single test result file and a single coverage report.

We need to add some configuration for test execution and coverage collection in the pyproject.toml:

[tool.pytest.ini_options]
addopts = """
    --import-mode=append
    --cov=my_project
    --cov-config=pyproject.toml
    --cov-report=
    """

[tool.coverage.paths]
# Maps coverage measured in site-packages to source files in src
source = ["src/", ".tox/*/lib/python*/site-packages/"]

[tool.coverage.html]
directory = "reports/coverage_html"

You can open reports/coverage_html/index.html to look at the coverage of your code.

Documentation

Proper documentation is essential for any high-quality Python project. It not only helps others understand and use your code but also assists you and your team in maintaining and extending the project over time.

Key elements of a good documentation are:

1. Project overview: Begin with a high-level overview of what your project does, its purpose, and its target audience.
2. Installation guide: Provide clear instructions on how to install and set up your project. Include any prerequisites, required dependencies, and steps for setting up a development environment.
3. Usage instructions: Explain how to use your project. Include code examples, use case scenarios, and any necessary configuration details.
4. API reference: If your project is a library or framework, include an API reference. Document each function, class, and method, explaining their purpose, parameters, return values, and any exceptions they might raise..
5. Contribution guidelines: Outline how others can contribute to your project. Include coding standards, the process for submitting pull requests, and guidelines for reporting issues.
6. Changelog: Maintain a changelog to document the history of changes, improvements, and fixes in each version of your project.

We will use Sphinx to build our documentation. Let’s add it to our docs extra in pyproject.toml:

doc = [
    "sphinx",
]

Run pip install -e .[dev] to make sure that Sphinx is installed into your virtual environment. We will not dive deep into all the ways in which a documentation can be structured and built. The Sphinx documentation has a lot of resources on how to build a nice documentation. The easiest way is to run sphinx-quickstart docs from your project root and follow the prompts. This will populate your docs folder with some basic folders and files to build a documentation. The conf.py is of special interest because it holds all the configuration for the documentation and we need to tweak how Sphinx retrieves the project’s version. Change the line starting with release = to:

release = importlib_metadata.version("my-project")

Now we just need to add the sphinx-build command to the docs step in our tox.ini:

commands =
    sphinx-build -b html -d "docs/build/doctrees" "docs" "docs/build/html"

tox run -e docs now generates HTML output for your documentation. There is more to writing a good documentation and keeping it up-to-date than can be covered in this guide, but the presented structure can be easily extended to include doctests (sphinx-build -b doctest ... ) or auto-generated API docs (sphinx-apidoc ).

Building the package

The last puzzle piece in our pipeline is to build a wheel and sdist of the project. These build artifacts can be distributed via a package index like PyPI or you can send them directly to your co-worker for installation.

We will use build to build our package and need to add it to the build extra in the pyproject.toml first:

build = [
    "build[virtualenv]==1.0.3",
]

You can now execute tox run -e build to build the package. The wheel and sdist can be found in the dist folder afterwards. As easy as that ;)

Conclusion

Congratulations! By following this guide, you’ve laid the groundwork for a robust and high-quality Python project as we head into 2024. Let’s recap the crucial elements that will set your project up for success:

1. Well-defined project structure: Adopting the “src layout” provides a clean separation of source code, tests, documentation, and configuration files.
2. Making the project installable: Using setuptools and setuptools_scm, you've made your project easily installable and manageable with pip. This step is fundamental for both development convenience and distribution.
3. Efficient pipeline setup: Implementing a pipeline with tox ensures that linting, testing, documentation, and package building are automated and consistent across environments. This setup boosts productivity and code quality.
4. Rigorous linting and static analysis: Leveraging tools like Ruff and Mypy for linting and type checking keeps your codebase clean, error-free, and maintainable. This proactive approach to catching issues early saves time and effort in the long run.
5. Comprehensive testing: By integrating pytest and measuring code coverage, you're ensuring that your code is robust and behaves as expected. This not only catches bugs early but also provides a safety net for future changes.
6. Thorough documentation: Using Sphinx for documentation maintains a clear and informative guide for users and contributors. Good documentation is invaluable for project adoption and maintenance.
7. Streamlined package building: With the configuration in place to build wheel and sdist packages, distributing your project becomes a breeze, whether it’s for private/internal use or for sharing with the world on platforms like PyPI.

You can follow this guide step-by-step to bootstrap your project or you can use a template to quickly generate a project. The guide is based on this template, so feel free to use it to your liking. I developed the general project structure during my time at voraus robotik and it is being used successfully every day for dozens of different projects.

A big benefit of the pipeline described in this guide is that it can be executed locally. Integration into a hosted CI system like GitHub or Jenkins is very straightforward via the tox run -e xxx calls. Adding a CI system and publishing the package to a package registry will be covered in a future blog post, so stay tuned.

Remember, this guide is a starting point. Every project is unique, and you may need to adjust or expand upon these foundations to suit your specific needs. As the Python ecosystem continues to evolve, stay curious and adaptable. Happy coding in 2024 and beyond! 🐍🚀