Yoyo moves to Python 3
At Yoyo Wallet we build and power payment, loyalty, and marketing tools to provide best-in-class results for retailers and beautiful experiences for their customers.
We use Python as our main programming language for all our backend services using well-known tools like Django and Celery. Thanks to our distributed infrastructure, we can move faster with quick deploys having robust and very responsive services.
Also, we have a strong culture to use data as a driver so we’re continuously analysing our data with scientific tools like scikit-learn, Pandas, NumPy and other more specific tools for visualisation and data processing.
Why did we decide to migrate?
It was August of 2018 and many Yoyo services were still using Python 2.7. We knew that we needed to think of a good strategy to migrate to Python 3 since Python 2 was going to be deprecated at the end of 2019.
Also there are other important changes which require special planning for doing the migration, to name a few:
- Python 3 is backwards-incompatible
- Integers division result type is float
- Implied relative imports are invalid
- Exception no longer have a message attribute
We started discussing and comparing strategies for migrating our services. We immediately defined the most important requirement; to be able to write code that would be compatible with both versions of Python. This was critical since we knew only code written before the new forward-compatible code would require updates.
Strategy
After discussing some proposals, we agreed on a good strategy to do the migration which could be summarised in the following clear steps:
1. Upgrade dependencies to be compatible with Python 3
2. Migrate individual Django apps or modules using Modernize. We discarded the idea of using tools like 2to3 since they can potentially migrate code to be compatible only with Python 3.
Following is an example of how we used to do migrations of individual modules:
3. Enable CI step to run linters to check code incompatible with Python 3 and make this step required just for the app or module migrated, this would guarantee that all new code would be forward compatible.
Example of how we setup the CI process to check for incompatible Python 3 code:
This configuration was updated for each module that was migrated.
4. Repeat steps 2 and 3 until the whole codebase is migrated.
5. Test services using Python 3.
6. Fix code incompatible with Python 3.
7. Deploy to pre-production environment.
8. Deploy to production.
The main advantage of this strategy was that the migration was mostly transparent to the team who could continue writing code compatible with Python 2 and 3 at the same time. This meant that the transition to Python 3 was never a blocker for our daily work and allowing us to write code in a good maintainable way.
Sometimes migrating code wasn’t easy as existing tests would fail. We needed to update the code to pass existing tests and add extra coverage for special cases around some of the major changes in Python 3. Examples of this include the use of unicode vs strings and object serialisation using pickle which is extensively used on Celery.
It is worth mentioning that the official documentation was very helpful and fundamental on taking decisions around which tools and how to do the transition.
Lastly, it is important to mention that having all our services running on Docker containers gave us great flexibility and made the transition and testing of Python 3 straightforward. We had the opportunity to replicate our production environment and do intensive testing before deploying the new version of Python.
Lessons learned, problems and solutions
Although the migration strategy worked very well and the transition to Python 3 was almost painless, there were some breaking changes that broke some of our services and dependencies (some could be updated but others needed to be replaced or removed completely), to name a few of the issues:
- Unicode vs string/bytes
- Relative imports
- Use of round
- Celery and pickle different between Python 2 and 3
The communication within the Engineering team was very important. First we communicated to everyone the importance and benefits of the migration and why everyone should focus on writing Python 3 compatible code. We also wrote handy documentation for everyone as a guide of advice on decisions when writing using Python 3 which ultimately allowed software developers to work more independently following conventions that we agreed on.
Having a good suite of tests is a key element on doing migration of this magnitude. At Yoyo, we encourage developers to make testable and reliable code writing unit and functional tests which gave us the confidence that the migration was going well and was going to work on production.
The deployment plan for the migration was simple since all the services were tested before moving to Python 3 in production. Having proper monitoring tools for all our services allowed us to track key metrics so we knew that everything was going well. One of the main difficulties we found was the change on serialisation using pickle and the incompatibility on Celery which led to some tasks failing. This required manual intervention where we replicated the failed tasks using the same data. Although the solution wasn’t straightforward, thanks to having an error tracking system in place that helped us to find issues in real time it was easier to monitor and get all the required information to replicate the tasks.
As an example of our monitoring, we expected to see an improvement on workers’ memory consumption, however the immediate effect was incredibly good and the long-term consumption was reduced by approximately 10% as shown in the following graph:
Since we used Modernize, the tool added many references to the six library to make code Python 2/3 compatible. For this reason, after migrating our services to use the new version, we needed to do a cleanup of the code to replace the use of six by the native tools from the language.
Finally, our original plan was to migrate all projects to the last stable version available which at that moment was Python 3.7, however some libraries supported only up to Python 3.6. In the case of our main platform, we tried to use 3.7 but there were too many issues from dependencies or even non-working functionalities so we didn’t have any other option than sticking with version 3.6. On the other hand, we encourage to start new projects and migrate as much as possible to the last one.
Resources
Following are some important resources that we used in the process of migration that were very useful during the migration process:
- https://docs.python.org/3/howto/pyporting.html
- https://python-future.org/index.html
- https://portingguide.readthedocs.io/en/latest/
Summary
In summary, Yoyo developed software for three years using mostly Python 2 and thanks to the simple, clear migration steps and a lot of effort the main platform and some other projects were migrated 100% to Python 3 in approximately two months. I also want to highlight that the migration did not require downtime of our services and the idea of doing a migration in steps by migrating small parts of the code compatible with the new version and a crucial team effort were key for this to work.
Special thanks to Gary Evans, jianyuan and Mehdi Shahinmehr for reviewing and giving feedback for this article.
