Making a Python library — how the ecosystem changed in 2.5 years

I created my first python library in August 2016, mostly to learn how to make one — not just writing setup.py, but also to have docs to maintain, possibly users to make happy, crafting a README — everything that comes with the territory.

Maybe there were better ways back then, but let’s establish it as a baseline. Here’s a screenshot of the repository’s file layout and top of the README:

Hopefully nothing far too surprising. Docs, examples, tests, some static analysis, packaging, dependencies, CI for supporting multiple python versions, setup.py bdist upload…

This is what I was ultimately happy with, however getting there was needlessly painful.

Recently I finally managed to find a reason to create another package (let’s ignore some abandoned projects and non opensource stuff). Compared to the first time, I now had not only previous experience, but I also had news to play with, that I wanted to test out. I was looking forward to leveraging PEP 518 (pyproject.toml) to build a package without most of the hassle, with a CLI to help me along the way. Here’s how it went.

Poetry

I have already replaced requirements.txt everywhere with Pipenv, but if you’ve been following the tools a little bit, you know that Pipenv is application only and doesn’t support multiple Python versions (at least not without hacking around a little bit) and Poetry was always named as it’s biggest competition, so I was really looking forward to trying it out and it did not disappoint. I am not a fan of the default file layout it generates for you, but if it ain’t broke… And it wasn’t, everything worked. Poetry is a huge improvement and makes it way easier to create your first python package.

Conclusion: BETTER

However, PEP 518 (and/or PEP 517) is not fully supported by the rest of the ecosystem just yet, which essentially meant quite a few things changed when it came to other tools.

Tox

whitelist_externals = poetry
skip_install = true
commands =
    poetry install -v
    poetry run pytest tests/

After looking up the solution you see above, tox wasn’t much of an issue, but it defintely doesn’t feel right. Thankfully next pip release already has a fix for this implemented!

Conclusion: SAME (in a couple of days)

Read the Docs

Unfortunately, until issue 4912 on their github is resolved, there is no way to build a documentation of a python package without requirements.txt and setup.py , so an additional dependency you have to remember to keep updated is needed. It’s called poetry-setup.

Oh and you know how there’s new Python versions, would you like to build your docs under python3.7, you know as Sphinx requires you to actually import the code? Only with conda. What about 3.6? Let me introduce you to .readthedocs.yml which you probably never knew even existed despite using RtD for a long time. You’ll need to put inside this:

build:
  image: latest
python:
  version: 3.6
  setup_py_install: true

Conclusion: WORSE

Travis CI

Of course you’d like your CI to run tests on the newest Python version, perhaps more than on some old version. Well, you can, but it’s not as simple as adding 3.7 to the list of supported pythons in your configuration yaml file. You have to be explicit about linux distro and require sudo in the job: https://github.com/travis-ci/travis-ci/issues/9069#issuecomment-425720905

Just a fun fact, Xenial is not even the newest LTS of Ubuntu, so expect this explicit dist requirement to eventually break, possibly before April 2021 when it’s LTS support ends.

Conclusion: WORSE

Sphinx

The golden standard of writing python docs. Maybe it’s just me, but Sphinx isn’t aging well. The problem is when I wrote this down as a note, I didn’t write down what exactly made me feel that way and if I can guess what it was. Perhaps it was type hints not really being natively supported. Or the overall feel of a tool that isn’t made for this decade (darn you my millennial-ish brain). Maybe it was me really wanting a theme that supports both light and dark theme. Possibly just me being bad with autodoc. And then there’s the possibility that I just see Read the Docs and Sphinx as a single tool and I’m just taking it out on Sphinx.

My point is, docs are hard. And it didn’t get any better. At least not with Sphinx.

Conclusion: WORSE (or at least my notes said so, for whatever reason)

Reno

This is an odd one, I bet the vast majority never heard of Reno and I wouldn’t either if I didn’t attend a talk about it at EuroPython 2018. Reno stands for Release Notes and it’s part of OpenStack, in a way.

In my previous package, there’s no changelog. And changelogs are cool — or at lest I like reading them. Reno makes it… Not so painful? Make kind of sense? I recommend watching the talk about it if you’re interested in learning how it works, I’ll just relay my experience. Huge tl;dw is: Through git tags and history it keeps track of special generated yaml files that have lists of ReST text under different keys — this can then be nicely integrated with Sphinx. And that’s why I didn’t experiment with Sphinx alternative this time — I really wanted to try it out. And it was good. Apart from writing ReST inside of YAML file. There are some quirks, but I quite enjoy maintaining a changelog now.

Conclusion: COOL

And that’s about it. If you are wondering what package I experienced this all with, it’s called pyArtifact (how original and totally not ambiguous! 🤫). Thanks for reading!