Oh no! This package is Python 2 only
You’re head down, working on a new project and one of your dependencies still doesn’t support Python 3 — argh!
Here’s a quick guide on how to solve that problem, step by step.
Our example package is a client library for Qualys (a security product)
Step 0: Check that nobody else has solved this
Before you fork the project, check that nobody else has hit this issue before, is there an issue in GitHub? Check in PyPi that another package doesn’t exist with the updates for Python 3.
Step 1: Fork it
When you start, you’ll need to fork the original project. Normally, at the bottom of the package details on PyPi you can see supported Python versions. It’s quite typical for people to omit this information for Python 2 only packages.
Go to the GitHub project and create your fork, download and clone the package. Create a branch straight away before you start making changes
Step 2: Print statements
The first thing you’ll typically hit before you start worrying about imports is just the simple print statements. Print statements without parenthesis is the most common Python 3 “unfriendly” hurdle I come across. We are going to use the Python-Modernize package in Step 3, but this won’t catch README files, documentation and code examples.
fgrep -r “print ” .
Ok, we have 18 occurrences in our package, including some in ./.git/ (which we ignore. Go into your favourite editor and fix all of those up.
Then check that lot in to a single commit git commit -a -m “print to python 3”
Step 3: Tests
The “best” way to test Python 2 and 3 compatibility is through complete testing and a CI service like Circle, Snap or Travis.
But guess what. Your package doesn’t have any tests. D’oh! You have 2 choices — 1) write tests, be a good citizen 2) do some static analysis
Static Analysis
Install python Modernize with pip install modernize and run it in info mode first python-modernize qualysapi , read through the output and get a general sense of what it wants to change. Look OK? Run those changes
python-modernize -wn qualysapi then commit those changes in another check in.
Adding tests
The most basic test is can I import the module/package/submodule? This catches a lot of Python 3 issues around namespaces. Look at the Tox project for running tests against multiple environments.
The second check you want to look at is whether this project does any string processing. There’s more than I can detail in this blog post about Python 3 strings, but look at the six project which covers off the “is it a string” problem and also the use of the unicode and byte conversion methods.
The third is going to require some experience with the package. You can’t write unit tests from an existing code base when you don’t know whether it currently works! Focus on integration testing and work backward. Try writing a simple script that uses the package as it was intended. Then build some tests around those scenarios.
Step 4: Update setup.py
Check out setup.py to see what changes need to be made to tell the world this package supports Python 3.
Ok, some of my pet peeves appear here:
- It says BSD license, and Apache 2?! But in the code files it says Apache 2, and in the directory there is no LICENSE file. This is really common and you have a problem since it essentially isn’t licensed properly on either. I would assume the setup.py is a copy+paste mistake and update it here, then add the LICENSE file.
- Markdown doesn’t render in pypi (as per screenshot). You can either live with this or change it to an RST file. Keep in mind this isn’t your project, so be nice!
- It doesn’t say which Python versions it supports. So let’s add that too
- Using Python-Modernize means the package has a runtime dependency on the six package. Add this to
install_requires.
See the install_requires field at the bottom? Also check that those dependent packages also support Python 3.
Step 5: Install it into your original project
Ok, now back to the start- we were working on a project and got stuck with this package? It’s time to update that requirements.txt file with the forked copy. Make sure you push your changes to github first (git push origin python3). Pip allows you to not just install from PyPi but also from a Git repository directly. You can also specify the branch after the URL using the @ symbol. The egg= is used to telling Pip the package name.
-e git+https://github.com/tonybaloney/qualysapi@python3#egg=qualysapi
This is the place where you can continue running solid integration tests against the new version.
Step 6: Raise a pull request
Now, back in GitHub you’ll want to share your work with the author and the broader Python community.
- Be nice.
- Say that you’d like to use this package with Python 3
- Thank them for their hard work in making the package
- Explain what you did
Here is my example https://github.com/paragbaxi/qualysapi/pull/21
Wait for it to be merged and then hopefully you’ll see an update on PyPi.
Step 7: 3 months later?
So the package author didn’t merge your code. That’s ok, it happens. Often because people authored packages as part of their work and changed jobs. Go back to setup.py and give the package a new name (don’t change the packages field) like qualysapi-py3. Push it to PyPi and then update the pull request with a message saying the name of the new package. That way when the next person comes along looking for Python 3 support they’ll see your post.
