[PMPL — Milestone 3] Upgrade Python 2 to 3 and Django 1.11 to 2.2

7 min readDec 7, 2019

Introduction

Hi, I’m back with another coverage on my works at PMPL’s class project.
This week (Dec 1–6, 2019), most of us (as a class) are refactoring codes (again!) and increasing test coverage inside Kape, the project about internship match-making website I’ve told you earlier (see previous article!). But there are some underlying problem that almost missed from our focus, the technlogy stack are too outdated! So, in this week I choose to upgrade backend stack of Kape from Django 1.11 to 2.2 and from Python 2 to Python 3. And this article explains on how I upgrade the stack and what changes I made in the existing code to comply with.

Python 2’s support will come to near end and Django 1.11’s support too!

The Python Foundation officially stated that they will end Python 2.7 support at 1 January 2020 (https://www.python.org/doc/sunset-python-2/). Feels like this is the same energy as Windows XP 5 years ago (it ends at 2014), right? Some of the existing products, companies, or communities were reluctant to upgrade, and that’s what makes Python 2.7 being the longest supported Python version ever (someone built the Python release cycle chart here: https://pythonclock.org/). Well, upgrading the Python version from 2 to 3 is not as hard as upgrading from Windows XP to Windows 8.1 though… so I’ll cover on how to upgrade, here.

Same energy also applies for Django too! Django will end the support for Django 1.11 at April 2020, yet many people still use that! Well, after Django 1.11, Django Foundation starts to release new Django version every four months, and x.2 is their LTS version (example: Django 2.2). Currently, they even recently release Django 3.0 but it’s too premature so that some of our existing dependencies still didn’t support it yet (yeah, it released only few days ago!) Upgrading Django 1.11 to Django 2.2 is also very easy!

Steps on Upgrading the Stack

Requirements that I used

Here are the new requirements on Kape compared to old one. Now I used more flexible versioning in favor of strict ones. I allow upgrading packages as long as it’s in the same major version (e.g. Django, I don’t allow Python to upgrade it to major version 3 as it will break many unexpected changes).

appdirs>=1,<2 (before: 1.4.0, now: 1.4.3)
coverage
Django>=2.2,<3 (before: 1.11.17, now: 2.2.8)
django-filter>=2.2,<3 (before: 1.1.0, before: 2.2.0)
django-nose
django-rest-swagger
django-silk
django-webpack-loader>=0.4,<1 (before: 0.4.1, now: 0.6.0)
djangorestframework>=3,<4 (before: 3.5.4, now: 3.10.3)
gunicorn
packaging>=19,<20 (before: 16.8, now: 19.2)
psycopg2-binary>=2,<3 (before: 2.8.3, now: 2.8.4)
pyparsing>=2,<3 (before: 2.1.10, now: 2.4.5)
requests
requests-mock
six>=1,<2 (before: 1.10.0, now: 1.13.0)

Difference between Python 2 and Python 3

Yes, you can always read this for full coverage on converting Python 2 code into Python 3 → https://docs.python.org/3/howto/pyporting.html, and you can also read about the package that will automate this process → http://python-future.org/automatic_conversion.html. But the first one is lengthy and the second one, ummmm I think using that automation will break codes on this great scale of work (yes, it’s a class [or batch??] project). So, I think I’ll sum up things here:

At Python 2, printing to standard output is like print "hehe", "haha". In Python 3, it’s print("hehe", "haha"). That little change in parentheses can make you frustrated sometimes so make sure you’ve changed all Python 2 style printing in your code into Python 3 ones.
At Python 2, direct importing like import views_constants from a file views_constants in the same package (in this case it’s inside core package) is no longer supported somehow. So, change the import notation into import .views_constants, or better change into import core.views_constants.
Python 2's basestring type now becomes str in Python 3. Replace them if you have any type comparisons like isinstance.
If you want to read binary files, and you don’t want Python 3 to automatically decode it into UTF-8 or so, please use rb mode instead of r. rb mode will return bytes after reading a file.
Don’t forget to clean up your Python 2 virtual environment if there any, and replace it with Python 3 ones by using this command: python -m venv env. Also please clean up all .pyc and __pycache__ files that are previously generated by Python 2 as they will create .pyc is stale error.

Problems on our Django and Django REST Framework code

After upgrading Python and its package dependencies for this project, I found some problem when running tests or runserver at Django. Yes, it’s about migrations, new string type, and some deprecated functions. Here are those differences:

At every Django models that have relation fields like ForeignKey, OneToOneField, or ManyToManyField, if you didn’t define on_delete yet, define it with models.CASCADE. At Django 2.2, on_delete parameter becomes mandatory, and at Django 1.1 it defaults with models.CASCADE. For those who don’t know what it means,models.CASCADE means that DB will delete every object that relates to a recently deleted object via that particular column (field).
Replace @list_route( decorator with @action(detail=False, and @detail_route( decorator with @action(detail=True,. Django REST Framework deprecates list_route decorator and detail_route decorator, replacing them with action decorator since 3.8.
Please don’t do any makemigrations before you upgrade from Django 1.11 to Django 2.2. If there are any migrations that existed before upgrade, search for any byte strings (example: b'kucing') and replace them into regular strings (example: 'kucing'). Django will show up weird error if there are parameters that filled with byte strings, somehow.

Another Problem: django-nose Bug that Renders Model/Serializer Uncovered

nose is a great testing tool for Python, and it also has a plugin for Django called django-nose. Unlike other tools, django-nose can capture the standard output of each test case and show it up later when the test case fails, so debugging will be easier. nose also integrated with coverage to check our code’s line or branch coverage. Unfortunately, for Django there exists some bug that prevents django-nose from correctly showing the coverage for models and serializers on Django REST Framework. Even if I added tests for models and serializers, the coverage won’t get up significantly because of that bug.

My Journey to Solve This

So… I tried three approaches to solve that, and found the latest one is successful:

Firsly, I tried to usecoverage on GitLab CI while maintaining the django_nose settings. Unfortunately, it FAILED for both Windows and Linux, showing no gain in code coverage.

Secondly, I tried to import coverage library via manage.py and dropping --with-coverage setting for django_nose, based on the suggestion here: https://github.com/jazzband/django-nose/issues/180#issuecomment-93371418. It showed SUCCESS on Windows, gaining the overall backend code coverage to 90% from 78%, and models/serializers coverage no longer 0%. Unfortunately, when pushed to GitLab CI, it FAILED to give similiar result.

So, I’ve tried another one! I DROPPED django-nose from requirements and use the standard unittest library, with coverage import still in manage.py to maintain existing workflow. Here are the changes on manage.py that I made for this approach.

is_testing = 'test' in sys.argv and 'help' not in sys.argv
    if is_testing:
        import coverage
        try:
            from kape.settings import TEST_COVER_PACKAGE as sources
            cov = coverage.coverage(source=sources, omit=['*/tests/*'])
        except ImportError:
            cov = coverage.coverage(omit=['*/tests/*'])
        cov.erase()
        cov.start()execute_from_command_line(sys.argv)if is_testing:
    cov.stop()
    cov.save()
    cov.report()
    try:
        from kape.settings import TEST_COVERAGE_RESULT_HTML_DIR as html_dir
        cov.html_report(directory=html_dir)
    except ImportError:
        pass
    try:
        from kape.settings import TEST_COVERAGE_RESULT_XML as xml_file
        cov.xml_report(outfile=xml_file)
    except ImportError:
        pass

It showed SUCCESS on both Windows and Linux, gaining the overall backend code coverage to 90% from 78%, and models/serializers coverage no longer 0%. So, I use this approach!

The latest approach comes with the expense of losing stdout capturing for each tests (yeah we lose precious django-nose feature), making it harder to debug tests in general. To avoid that, in GitLab CI, I replaced python3 manage.py test with python3 manage.py test -v 2 to make the standard output seems like grouped for each test :) Yes, it is not as pretty as django-nose ones, but at least I hope it will help debugging easier.

Name                                  Stmts   Miss  Cover
---------------------------------------------------------
core/__init__.py                          0      0   100%
core/admin.py                            20      0   100%
core/apps.py                              7      1    86%
core/lib/__init__.py                      0      0   100%
core/lib/mixins.py                        6      0   100%
core/lib/permissions.py                 120     53    56%
core/lib/validators.py                   23      7    70%
core/migrations/0001_initial.py          10      0   100%
core/migrations/__init__.py               0      0   100%
core/models/__init__.py                   7      0   100%
core/models/accounts.py                 140     15    89%
core/models/feedbacks.py                 10      0   100%
core/models/recommendations.py           16      2    88%
core/models/vacancies.py                 57      0   100%
core/serializers/__init__.py              0      0   100%
core/serializers/accounts.py            125      2    98%
core/serializers/feedbacks.py             6      0   100%
core/serializers/recommendations.py       6      0   100%
core/serializers/vacancies.py            78      6    92%
core/views/__init__.py                    0      0   100%
core/views/accounts/__init__.py           6      0   100%
core/views/accounts/company.py           20      0   100%
core/views/accounts/login.py             54      0   100%
core/views/accounts/registration.py      29      0   100%
core/views/accounts/student.py           72     14    81%
core/views/accounts/supervisor.py         8      0   100%
core/views/accounts/user.py              21      2    90%
core/views/accounts/utils.py              7      0   100%
core/views/feedbacks.py                  52      0   100%
core/views/recommendations.py            52      6    88%
core/views/sso_login.py                  26      4    85%
core/views/vacancies.py                 399     37    91%
core/views/views_constants.py            68      0   100%
---------------------------------------------------------
TOTAL                                  1445    149    90%

The Final Result!

Here comes the result! The quality code before this were failed, indicating that code coverage must surpass 80%, and at that point it was 67%. Now, it’s jumped into 83.3%! So great!!

Quality Gate is now Passed! Coverage bumped from 67% to 82%!

That’s all from me right now, and it is the end of my PMPL Class Project blog coverage series~ Thank you for all of your attention and hope you’re getting some meaningful lessons out of it!

Greetings and have a nice semester, my golden-hearted friends!

Ichlasul Affan (from A̶u̶-̶P̶P̶L̶, ups…) — 1606895606

#PMPL2019 #̶I̶T̶’̶s̶T̶h̶e̶R̶e̶a̶l̶T̶h̶i̶n̶g̶