Upgrading to Python 3

Since Python 3’s initial release in 2008, many of us in the community have known that we’d have to update our production systems. A quick search for “Python 3 upgrade” provides plenty of guides outlining the differences between Python 2 and 3 — but we don’t see many guides for upgrading at an organisational level. With Django 2 just around the corner in the next couple of years, and most libraries supporting Python 3, now is the best time for development teams to get ahead of the curve. This guide will document the processes we instituted, what succeeded and what didn’t go so well. I hope that you can use this series to help manage your own upgrade to Python 3.

Like most organisations, we started building applications when Python 2 was the dominant language version (2011 in our case). Also, like most organisations, our primary application is one large software application in a single repository built with no regard for future versions of the language. More recently, we have built smaller, related, applications also in Python 2.

Purpose of this Series

Throughout this series, I’ll be covering the steps we took, how we got buy-in across the business, and what systems we used to help do this upgrade. I’ll go through some of the great successes and our biggest challenges. Finally, I will be reflecting on prior decisions and determine how effective they were.

My ultimate goal is that you can use this series to inform your own decisions when upgrading to Python 3.

Who are we?

Before I go into the detail of how and why we’re performing this upgrade, let’s get some context into who Pebble are and how we got here.

Pebble is a software company that has been around since 1992 serving the education sector with financial management tools. Around 2010, I joined part-time as a student and, upon graduating in 2011, joined the team full-time migrating our client-based application — written in xBase++ — to a web service using Django. At the time, Python 2.6 was still the most popular version and pip was nowhere near ready for production development.

Over the past 5 years, we further developed this system, improving functionality and integration with third parties. We also absorbed the functionality of our grants database (originally written in PHP) and migrated that into Django as well.

In terms of team size, we have fluctuated between just being myself (for large periods of time) to a team of 4 developers. As of writing this article, we comprise of three full-time developers with a full list of features to develop, in addition to maintaining the system uptime and keeping on top of software updates. Sounds like just the right time to initiate a major upgrade then!

Why Upgrade?

Given how small our team is and how long our job list is, why take on the responsibility of a major system upgrade like changing out the language underpinning your primary application?

Simple: because of our small size we’re not in a position to maintain the entire stack of all our applications by ourselves. We are almost completely dependent on the wider open source software community to help us provide the fantastic service and support that our customers expect of us. This is the practical side of our culture of engaging with open source whenever possible.

With Django 2.0 on the horizon in 2017/2018, it’s clear that Python 2 support is soon to be at an end and, with it, future features, bug fixes, and security patches provided by the amazing wider community.

Starting the Upgrade

How does a 4-man team go about upgrading a good 100,000+ lines of code to a new language? Well, there’s all the obvious things I could write about (tests, coverage, shim libraries) but there’s nothing really covering the organisational requirements. How does one bring the business around to work not directly related to feature development? How do we know when’s a good time to flip the switch? What do we need to do to minimise customer impact?

The Plan

The first people who need to buy into this is our own development team. The developers are the guys doing this, so we all need to be on board with this upgrade! We started by by calling a meeting with the following agenda:

1. Inventory our Python-based services
2. Agree a supported OS
3. Agree a Python version
4. Produce an initial upgrade timeline

Out outcomes were:

1. Our supported OS+Python version were Ubuntu 16.04 and Python 3.5.
2. We agreed our primary application’s server should be upgraded to Ubuntu 16.04 before January 2017
3. By December 2016, we should know what packages support Python 3 and which ones we need to switch out
4. We should implement coverage checking by the end of January 2017
5. All new projects are Python 3 only from January 2017
6. We should implement tox to test Python 2+3 simultaneously
7. We can test an upgrade process on our smallest project
8. We have a progress report at the start of every month

Selling it to the rest of the business

Now we have a plan and the development team have bought into the need for an upgrade, it’s time to let the rest of the business know.

Why is this important? Simple: we are effectively doing maintenance that may impact the speed at which we bring new features to market or we may even cause system instability. Explaining to the head of customer accounts why we’re doing this work is far far easier when she’s not in the middle of answering the phone to a customer whose day you’ve just ruined. Chances are good that your business has an entire customer service team there to help you manage fallout from issues exactly like this. Help them to help you deliver a great service!

How did we do so far?

We gave ourselves some deadlines for the start of this year, so it’s fair to start by evaluating ourselves early. Out of 10? I’d give us a 5 initially.

• Our primary application server is still on Ubuntu 14.04
• While we’ve updated our smallest app to Python 3, we have yet to get that build into production
• We haven’t implemented a coverage checker
• Our new applications are all Python 3
• Our code reviews are adding a focus on Python 3 incompatible constructs.
• We had our first monthly review

Our biggest feedback was leaving our initial review so long (our first meeting was in September). Using a monthly review and publishing posts like this, we are hoping to make ourselves more publicly accountable.

Feedback

I’d love feedback on our process. Have you done your own Python 3 upgrade and what advice would you like to share? Please feel free to either comment here, or tweet me @scott_walton and I’ll be happy to discuss!

Next Steps

In the next post, I’ll post a short review of the first couple of months, where our challenges lay, what could be preventing us moving forward and how we plan to overcome these issues.