A Ready Reckoner To Upgrade From Python 2 to Python 3
A touch upon how to migrate your environment and service codebases from Python 2 to Python 3.
Quite often developers face situations in which they need to upgrade their applications and related environment to newer versions of the respective stacks.
In the first path of this blog, I’ll focus on how an ubuntu 16.04
server running python 2.7 can be upgraded to boot the application with the python version 3.7
. In the second part, I discuss how the application codebase can be migrated from Python 2 to Python 3.
Migrating the server
These are some general steps that can be followed to perform the update from a Python 2 based environment to a Python 3 environment.
The first step is to add deadsnakes
PPA to the machine. This contains more recent Python versions packaged for Ubuntu.
add-apt-repository ppa:deadsnakes/ppa
Next, install the required python version — I'll be using version 3.7
for the example
apt update
apt install python3.7
After the successful execution of the last step, create a symlink to the python 3 installations, this is not something done by default by the system and will be handy in case your applications rely on using the python
command
If multiple python 3 installations are required, this might come in handy.
# different versions can be specified here and priorities be set
update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 2
After successfully creating the symlink, python 3 dev tools and python 3 pip installation is required to allow building and managing packages.
apt-get install python3-pip python3-dev
python3 -m pip install --upgrade pip
More specific python-dev tools can also be installed, to establish direct compatibility with the installed python version like this👇
apt-get install python3-pip python3.7-dev
python3.7 -m pip install --upgrade pip
As the last step, install the virtual environment for Python 3.
pip3 install virtualenv
The Automation
The above steps can be combined into a shell script, and be integrated with the application execution or can be executed manually on the machine.
#!/usr/bin/env bash
add-apt-repository ppa:deadsnakes/ppa -y
if [ $? -eq 0 ]; then
echo "PPA installed correctly"
else
echo "Failed to install PPA"
fi
apt update
if [ $? -eq 0 ]; then
echo "Apt Update Passed"
else
echo "Apt update failed"
fi
apt install python3.7 -y
update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 2
if [ $? -eq 0 ]; then
echo "Python version 3.7 installed. Use python3 --version to check that correct version is installed"
else
echo "Failed to install python version 3.7"
fi
apt-get install -y python3-pip python3.7-dev
python3.7 -m pip install --upgrade pip
if [ $? -eq 0 ]; then
echo "Python 3.7 pip and dev tools installed"
else
echo "Failed to install python 3.7 pip and dev tools"
fi
pip3 install virtualenv
if [ $? -eq 0 ]; then
echo "Installied pip3 virtaul environment"
else
echo "Failed to install pip3 virtual environment"
fi
pip3 install --ignore-installed PyYAML
if [ $? -eq 0 ]; then
echo "Successfully upgraded to python3"
else
echo "Failed to upgrade to python3"
fi
Migrating application codebase
There might be a number of tools to help with the automated migration of the application codebase, one of these is 2to3
, which can be used to modify the code to be compatible with python 3.
Here is how one can use this utility to modify code to be compatible with python 3.
pip install 2to3
cd <your project directory>
# first have a look at the modifications this will make
2to3 .
# to modify in place
2to3 -w .
Note: 2to3 will try to correct your imports to python 3 convention, which may not always be correct as it assumes the imports are from packages. So it’s good to check the modifications that 2to3 will make before applying them. More detail on this utility can be found in this official doc.
It can’t be that simple to migrate to Python 3, right..?
The above step will modify the imports and the usual statements like print
to their python 3 versions, but some other code regions like error handling, comparisons, and string handling may still need some work.
Let us look at these in more detail.
Integer Comparisons
So python 2 did not put a hard check on types for comparison so something like the below piece of code will actually work
a = "2"
if a >= 0:
print "True"
else:
print "False"
But the same will not work with python 3, so you need to modify the above to cast the string to an integer to make any such comparison.
a = "2"
if int(a) >= 0:
print("True")
else:
print("False")
Error Handling
The details about error handling in Python 3 can be read from the docs here. But on a very simple level, a modification of error logging or propagation may have to be modified. An example of this is presented below
Python 2try:
<code block>
except <Error> as exp:
print exp.messagePython 3
try:
<code block>
except <Error> as exp:
print(exp)
Difference between string and bytes
In python 2 the difference between string and bytes was not very well defined but coming to python 3 there is a clear distinction and they may not be used in place of one another.
What kind of problems could this lead to?
This can lead to a number of problems, as python 2 allowed free usage of string objects where a byte would have been expected and vice versa.
For instance, if you previously opened a file in a binary mode and tried to populate it with strings it would have worked in python 2 but the same will cease to function in python 3.
file = NamedTemporaryFile(mode="w+b", delete=False)
file.write("some string")
file.close()
The above piece of code works well with python 2 but will error out for python 3. You can modify it to the following so it may work
file = NamedTemporaryFile(mode="w+", delete=False)
file.write("some string")
file.close()
Similarly, if you were passing anything as a byte object where the expectation was for string the code will fail.
To handle this correctly for python 3 a modification like this should help
try:
x = x.decode("utf-8")
except (UnicodeDecodeError, AttributeError):
pass
Is there anything else that needs to be taken care of?
The issues faced while migrating any codebase to python 3 may not just be limited to the above scenarios, they may contract or expand based on the codebase in question. But the above-mentioned scenarios do cover a fair amount of issues that may occur during migration.
Also, some issues related to Method Order Resolution or (MRO)
might turn up. This comes into play when a class derives from multiple classes, this may also pose a challenge in the migration of codebase from python 2 to python 3.
Endnotes
By handling these scenarios one can successfully migrate their code to Python 3. One thing that I would mention here is having a good test suite setup will really be helpful in early detection and resolution of issues and also in gaining confidence over the health of the migration.