Goibibo walks the Python 3 path

More than a travel company Goibibo may also be called as a technology organization. We strive to use the best available technology stack in the industry and keep upgrading the stack and versions as and when we see a benefit. Working on newer technology, not only helps us to be on top of the cutting edge technology but also helps keep the engineers motivated and hone their skills. The below blog lists down the recent movement we made to Python 3 for our tech stack optimizations.

2to3 Utility:

Python 3 provides 2to3 utility which takes Python 2 code as input and transforms it to Python 3 code.

We took Python 2to3 utility doc as a reference and changed the code accordingly.

We faced numerous challenges while doing this transition, some of which are listed below.

1. Unsupported Pickle Version:

Pickle is a module in Python used to serialize and de-serialize a Python object. Python 3 uses protocol 4 as default pickle protocol. We are using Redis as Django cache backend. django-redis-cache uses pickle to dump the data into Redis and if we are using Python 3 then it will use pickle protocol 4. Python 2.7 supports highest pickle protocol 2. This caused an issue when we tried to load data (dumped into Redis using protocol 4) from Python 2.7.

For this we created custom Redis backend library for Django and mentioned pickle protocol as 2 in the set method of CacheClass as below :

self._set(key, pickle.dumps(value,protocol=2), int(timeout), client)

2. Django Rest Swagger:

We are using django-rest-swagger to document our APIs.

In Python 3, version 0.3.x of django-rest-swagger is not supported anymore. So we updated it to latest 2.1 version.

Since YAML docstrings are deprecated in django-rest-swagger 2.1, there is no way to mention post-params using YAML docstrings.

So I took help from following StackOverflow answer for this

This version of django-rest-swagger uses rest_framework.schemas.SchemaGenerator to generate the schema and SchemaGenerator uses get_serializer_fields to get the serializer information of a view. get_serializer_fields checks if a view has a get_serializer method to generate the form.

So we used GenericAPIView class of django-rest-framework instead of APIView. We created custom serializer to define all parameters of a POST request.

Because of this, we could easily override serializer_class attribute of GenericAPIView class in our API and used below settings in our settings.py file

SWAGGER_SETTINGS = {
'VALIDATOR_URL': None,
'JSON_EDITOR': True
}

This is how swagger doc looks in browser

Swagger Docs

3. Eticket PDF:

Earlier we were using Pisa for creating pdf from the template. As Pisa is not supported in Python 3. We used xhtml2pdf for creating pdf instead of Pisa in Python 3. xhtml2pdf also provides line callback for generating images on pdf. Normally xhtml2pdf expects these files to be found on the local drive. They may also be referenced relative to the original document. But we wanted to load from different sources like the Internet via HTTP requests or from a database or anything else. Therefore we defined a line_callback that handles these requests.

4. NewRelic Python Package:

NewRelic is a performance monitoring tool. We were using an older version of newrelic python package (2.64.0.48) and after upgrading to Python 3, it stopped reporting the internal details of the APIs. We upgraded the latest version of the package (2.88.1.73) and it worked fine with Python 3.

You can refer below StackOverflow documentation also if you find any other issues.

Some small code snippets to keep in mind when migrating:

+----------------------+------------------------+
| Python 2.7 | Python 3.6 |
+----------------------+------------------------+
| except Exception, e: | except Exception as e: |
| xrange | range |
| iteritems | items |
| has_key | in |
| print "hello" | print("hello") |
| unicode | str |
| import urlparse | import urllib.parse |
+----------------------+------------------------+

Conclusion:

After upgradation to Python 3 and Django 1.11, We got the significant reduction in CPU uses and memory uses. We got slight improvement in response time of the Application.