Taming the beast: Python Package Maintenance Made Simple

How to reduce package maintenance like a pro

If you have ever created a Python package, you probably started out with a nice and tidy state: All dependencies up-to-date, bunch of useful metadata and your tooling state of the art. But also very likely, after a while you realise that you have to update some dependencies — and/or you lack a bunch of neat meta information on PyPI.

After creating the n-th package, it felt so imensely un-DRY to have the same configuration files over and over again. Often, I would find a new linter and/or new metadata settings which I definitely want to use in all of my packages. As a result, I had to change every single one of them. Tedious.

Fun fact: I’m not the only one struggling with this topic. In the well-known Django Chat community-famous Adam Johnson talks about something very similar.

To tackle this issue, I created a meta package called the Ambient Package Updater. This package provides a CLI command to automatically update your favourite packages metadata and dependencies. Furthermore, it keeps you sane by taking care of relations: If you drop an old Python version, this needs to be reflected in multiple files, e.g. the test matrix.

But let’s not get ahead of ourselves…

Photo by Kira auf der Heide on Unsplash

USPs

So why would I want to use this solution?

• Let somebody else worry about the latest and nicest meta data for your package
• Let the updater update all your configuration for all of your packages
• 97% of all the configuration files could be the same for each of your packages without you reinventing the wheel per package that you maintain
• You can view the changes in git and decide what to commit and if some additional changes are required
• Standardised process instead of fiddling around in every single package
• GitHub test matrix to validate that your package works with all the versions specified
• Awesome and standardised linters (ruff, pyupgrade, django-upgrade) and code formatters (black) already in place

How it works

Luckily, that’s really easy. Just navigate to your package in your favourite CLI, add ambient-package-update as a dependency and install it.

pip install -U ambient-package-update

Now you can use the updater to rewrite your configuration files

python -m ambient_package_update.cli render-templates

That’s it! Selected files containing relevant metadata have been rewritten. Now, you can check the changes via git, add or change as you desire and finally make a new release of your package. Neat, right?

Requirements

Now you’re probably thinking: “There have to be some things I have to do before I can use this super neat updater?” — and you are right, of course.

These things need to be done before the updater can work it’s magic.

• Metadata: Add a python file which describes your package
• Structure: Your project has to be structured in a certain (straight-forward) way
• Documentation: Your documenation (which your package of course has!) has to follow some simple guidelines

Metadata

You have to have a single Python file which contains your packages metadata.

# .ambient-package-update/metadata.py

from ambient_package_update.metadata.author import PackageAuthor
from ambient_package_update.metadata.constants import DEV_DEPENDENCIES
from ambient_package_update.metadata.package import PackageMetadata
from ambient_package_update.metadata.readme import ReadmeContent

METADATA = PackageMetadata(
    package_name="my_fancy_package_name",
    authors=[
        PackageAuthor(
            name="Albertus Magnus",
            email="albertus.magnus@ambient.digital",
        ),
        PackageAuthor(
            name="Thomas von Aquin",
            email="thomas.von.aquin@ambient.digital",
        ),
    ],
    development_status="5 - Production/Stable",
    readme_content=ReadmeContent(
        tagline="This package is great and solves problem X",
        content="""
# I am a very neat markdown text which helps people understand and use my package.
""",
    ),
    dependencies=[
        "django >=3.2",
    ],
    optional_dependencies={
        "dev": [
            *DEV_DEPENDENCIES,
        ],
    },
    ruff_ignore_list=[],
)

To help you keep your sanity, all those classes are dataclasses with type hints so you get lots of hints on how to populate those classes.

The file lives on root level of your project under .ambient-package-update/metadata.py. You add your package name, list your authors, describe which development status your package is in, add a readme as a markdown string and finally add dependencies.

To avoid having any user of your package install all the stuff that you need to implement your package, it’s wise to separate your dev dependencies from the regular ones. If you are in need of having more extras, feel free to add as many as you like.

Finally, you get the chance to opt-out on some ruff linting rules.

Structure

Once you set up your metadata as described in the previous paragraph, you have to have only a couple of directories.

Starting with docs/, this will contain your Sphinx documentation for readthedocs.org. Secondly, your package code has to live on the root level, in the example below it’s django_dynamic_admin_forms/.

Example for a package that’s compatible with the updater

Lastly, the testapp/ contains the Python test files. You can use either unittest or pytest. The pytest runner shipped from the updater can run both.

All the other marked files are being generated for you — no need to worry about them.

Documentation

This directory contains a standard Sphinx setup to be published at readthedocs.org — the most well-known space for open-source documentation. It has a neat integration with GitHub so that you can rebuild the docs on a git commit.

Outlook

At this point you surely are fully convinced that the solution is awesome and you are chomping at the bit to get started. Naturally, as to nearly all technical solutions, there are a few issues we haven’t tackled yet.

Dependencies

Even though you can define dependencies dynamically via the metadata.py file and add as many extras as you wish, the whole package will still be dependent on Python and Django. While you’d have a hard time building a Python package not dependent on Python, you might not want to have Django as a dependency.

Test matrix

As pointed out, we have and automatically create a test matrix for GitHub which tests every supported Python/Django combination. If you are using a different git provider like GitLab, you have to add your tests manually.

Even though this might lead to a huge number of test cases, in theory you should test your package also with all supported versions of your dependencies. For example, your package does some magic with images and requires therefore the library Pillow, then you should declare all compatible versions of Pillow and add those to your test matrix.

Sadly, this is not supported by default but you can add the behaviour manuall after running the update command.

Photo by Denys Nevozhai on Unsplash

Conclusion

If you are not using GitHub and are not depending on Django, the current version might not be the best fit — even though it would still add some neat metadata for PyPI.

In a nutshell, if you are creating (or maintaining) a Python package which depends on Django, checking out the Ambient package updater can save you a lot of repetetive tasks and time.

As common in the Django-ecosystem, this tool is opinionated in the way which linters and code-formatters we have chosen. If you feel that we missed something important, please come forward and open an issue or PR at GitHub. If, on the other hand, you like the idea but are not happy with one of the choices made, you can easily fork the repo and adjust as much as you like!