A Ready Reckoner To Upgrade From Python 2 to Python 3

A touch upon how to migrate your environment and service codebases from Python 2 to Python 3.

Python 2 to Python 3

Quite often developers face situations in which they need to upgrade their applications and related environment to newer versions of the respective stacks.
In the first path of this blog, I’ll focus on how an ubuntu 16.04server running python 2.7 can be upgraded to boot the application with the python version 3.7. In the second part, I discuss how the application codebase can be migrated from Python 2 to Python 3.

Migrating the server

These are some general steps that can be followed to perform the update from a Python 2 based environment to a Python 3 environment.

The first step is to add deadsnakes PPA to the machine. This contains more recent Python versions packaged for Ubuntu.

add-apt-repository ppa:deadsnakes/ppa 

Next, install the required python version — I'll be using version 3.7 for the example

apt update
apt install python3.7

After the successful execution of the last step, create a symlink to the python 3 installations, this is not something done by default by the system and will be handy in case your applications rely on using the python command

If multiple python 3 installations are required, this might come in handy.

# different versions can be specified here and priorities be set 
update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 2

After successfully creating the symlink, python 3 dev tools and python 3 pip installation is required to allow building and managing packages.

apt-get install python3-pip python3-dev
python3 -m pip install --upgrade pip

More specific python-dev tools can also be installed, to establish direct compatibility with the installed python version like this👇

apt-get install python3-pip python3.7-dev
python3.7 -m pip install --upgrade pip

As the last step, install the virtual environment for Python 3.

pip3 install virtualenv

The Automation

The above steps can be combined into a shell script, and be integrated with the application execution or can be executed manually on the machine.

#!/usr/bin/env bash

add-apt-repository ppa:deadsnakes/ppa -y
if [ $? -eq 0 ]; then
    echo "PPA installed correctly"
else
    echo "Failed to install PPA"
fi
apt update
if [ $? -eq 0 ]; then
    echo "Apt Update Passed"
else
    echo "Apt update failed"
fi
apt install python3.7 -y
update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 2
if [ $? -eq 0 ]; then
    echo "Python version 3.7 installed. Use python3 --version to check that correct version is installed"
else
    echo "Failed to install python version 3.7"
fi
apt-get install -y python3-pip python3.7-dev
python3.7 -m pip install --upgrade pip
if [ $? -eq 0 ]; then
    echo "Python 3.7 pip and dev tools installed"
else
    echo "Failed to install python 3.7 pip and dev tools"
fi
pip3 install virtualenv
if [ $? -eq 0 ]; then
    echo "Installied pip3 virtaul environment"
else
    echo "Failed to install pip3 virtual environment"
fi
pip3 install --ignore-installed PyYAML
if [ $? -eq 0 ]; then
    echo "Successfully upgraded to python3"
else
    echo "Failed to upgrade to python3"
fi

Migrating application codebase

There might be a number of tools to help with the automated migration of the application codebase, one of these is 2to3 , which can be used to modify the code to be compatible with python 3.

Here is how one can use this utility to modify code to be compatible with python 3.

pip install 2to3
cd <your project directory>
# first have a look at the modifications this will make
2to3 .
# to modify in place
2to3 -w .

Note: 2to3 will try to correct your imports to python 3 convention, which may not always be correct as it assumes the imports are from packages. So it’s good to check the modifications that 2to3 will make before applying them. More detail on this utility can be found in this official doc.

It can’t be that simple to migrate to Python 3, right..?

The above step will modify the imports and the usual statements like print to their python 3 versions, but some other code regions like error handling, comparisons, and string handling may still need some work.

Let us look at these in more detail.

Integer Comparisons

So python 2 did not put a hard check on types for comparison so something like the below piece of code will actually work

a = "2"
if a >= 0:
  print "True"
else:
  print "False"

But the same will not work with python 3, so you need to modify the above to cast the string to an integer to make any such comparison.

a = "2"
if int(a) >= 0:
  print("True")
else:
  print("False")

Error Handling

The details about error handling in Python 3 can be read from the docs here. But on a very simple level, a modification of error logging or propagation may have to be modified. An example of this is presented below

Python 2
try:
  <code block>
except <Error> as exp:
  print exp.message
Python 3
try:
  <code block>
except <Error> as exp:
  print(exp)

Difference between string and bytes

In python 2 the difference between string and bytes was not very well defined but coming to python 3 there is a clear distinction and they may not be used in place of one another.

What kind of problems could this lead to?

This can lead to a number of problems, as python 2 allowed free usage of string objects where a byte would have been expected and vice versa.

For instance, if you previously opened a file in a binary mode and tried to populate it with strings it would have worked in python 2 but the same will cease to function in python 3.

file = NamedTemporaryFile(mode="w+b", delete=False)
file.write("some string")
file.close()

The above piece of code works well with python 2 but will error out for python 3. You can modify it to the following so it may work

file = NamedTemporaryFile(mode="w+", delete=False)
file.write("some string")
file.close()

Similarly, if you were passing anything as a byte object where the expectation was for string the code will fail.

To handle this correctly for python 3 a modification like this should help

try:
  x = x.decode("utf-8")
except (UnicodeDecodeError, AttributeError):
  pass

Is there anything else that needs to be taken care of?

The issues faced while migrating any codebase to python 3 may not just be limited to the above scenarios, they may contract or expand based on the codebase in question. But the above-mentioned scenarios do cover a fair amount of issues that may occur during migration.

Also, some issues related to Method Order Resolution or (MRO) might turn up. This comes into play when a class derives from multiple classes, this may also pose a challenge in the migration of codebase from python 2 to python 3.

Endnotes

By handling these scenarios one can successfully migrate their code to Python 3. One thing that I would mention here is having a good test suite setup will really be helpful in early detection and resolution of issues and also in gaining confidence over the health of the migration.