Converting process of Python 2 project to Python 3

You can start reading on this but to know about my base story refer to this.

There are a few things you need to know before jumping straight to conversion.

1. Make sure you have good test coverage (Our current project uses pytest testing framework for this. Reference)
2. Learn the differences between Python 2 & 3 (Reference)
3. Use Pylint to help make sure you don’t introduce linting mistakes
4. Make sure any python library used is compatible with Python 3 as well
5. Finally, run the 2to3 Python auto-translation tool on the codebase

1. Make sure unit tests are written (Pytest)

First, make sure the project has unit tests written at least for main core components. If not start writing unit tests, otherwise there’s no way of verifying all the components and functions are working as intended after the Python 3 migration. You may think it as a burden but trust me this will definitely help you, writing unit tests is fun and one of the most important qualities that all software engineers should adapt to their work life. Since this is a crucial Python component that runs in production and any anomalies or bugs should be introduced with the migration this initial step is highly needed.

2. First, learn the differences in Python 3.

Without blindly using tools for conversion always try to get the knowledge and pass that knowledge to other developers as well. You can refer to so many online materials on this. I recommend this. With my experience in Python 3 migration, I faced a lot of issues regarding Python 3.0 using the concepts of text and (binary) data instead of Unicode strings and 8-bit strings. And also some module names were changed and I had to change them. For example, see the following table. Don’t worry you don’t need to go through all the code base module imports, we can get the help of Python 3 pylint and 2to3 tools. They will notify you about those changes automatically. I’ll later explain bit more.

┌────────────────────┬──────────────┐
│      Old Name      │   New Name   │
├────────────────────┼──────────────┤
│ _winreg            │ winreg       │
│ ConfigParser       │ configparser │
│ copy_reg           │ copyreg      │
│ Queue              │ queue        │
│ SocketServer       │ socketserver │
│ markupbase         │ _markupbase  │
│ repr               │ reprlib      │
│ test.test_support  │ test.support │
└────────────────────┴──────────────┘

3. Setup pylint for Python 3 and set up the local dev environment

Before starting any conversion you need to set up your local environment to suit Python 3. I use Ubuntu 18.04 OS and VS code as the IDE for my developments. In Ubuntu 18.04 is Python 3 already there. But if you use Windows or any other OS first install Python 3 interpreter first. But pip3 is not available out of the box. So we need to install it. Pip3 is the package management system for Python3 artifacts.

Installing pip3 (globally in your OS)

Downloading get-pip.py file for pip installation

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

To install pip3 run the downloaded get-pip.py file using Python 3 interpreter.

python3 get-pip.py

Now that you’ve installed pip3 let’s set up virtual environments for the development. It’s always good to create and work on Python virtual environments without spoiling the OS.

Create a virtual environment for development.

pip3 install virtualenv
python3 virtualenv python3-venv

The name of the virtual environment that I gave was python3-venv.

Now let’s activate the virtual environment.

. python3-venv/bin/activate

Note → Yes there’s a dot in the above command.

Now since you have activated the virtual environment your default Python interpreted and pip is set for Python 3. To check it just type python -V, and pip -V.

Now you can install pylint in the virtual environment using pip.

pip install pylint

Running pylint on the entire code base. Created a simple bash script called pyilnt.sh

#!/usr/bin/env bash
find [enter_root_module]/ -iname "*.py" | xargs pylint $1 $2 $3

You can use your root level directory name in the script and it will recursively find all the .py files in the entire project.

Running pylint command

./pylint.sh --disable=R,C,W

to disable all the Warning, Convention, and Refactoring messages. I right now consider only Errors. This will output the results as below. (I only include sample outputs to give you an idea)

Now that we have an idea where our code breaks in Python 3. We’ll next run 2to3 tool to see the output.

Installing the 2to3 tool and running 2to3

sudo apt-get install -y 2to3

Now we’ll create a script to run 2to3 inspection for the whole codebase. Create a file named 2to3_convertor.sh

#!/usr/bin/env bash
find [enter_root_module]/ -iname "*.py" | xargs 2to3

Run 2to3

./2to3_convertor.sh

Now let’s see the output. I’m not going to share all the details here. But get an idea about the output.

Note → here I’m not auto fixing the errors. It’s better for the developer to manually go through all the errors and manually fix the code. If you want to auto fix the code you can pass -w flag to 2to3 tool and allow the tool to fix the code. Refer 2to3 man page for more info.

After fixing all the code now it’s the time to test everything work as expected.

Now comes the importance of unit tests. If you run units tests again it should not fail.

In the next article, I’ll share how I test the agent in the Ubuntu OS not locally we cannot test it in local computers. The agent works only on the remote devices which run on Ubuntu 14,04 and 18.04

Next article → https://medium.com/@chamaral/how-the-python-3-upgrade-happened-remotely-to-all-those-ubuntu-devices-ab160809046b