Building and deploying an Enterprise Django Web App in 16 hours

Published in

Python Pandemonium

12 min readApr 9, 2018

A week ago I put together an application for internal employees to be recognised for learning achievements and share them with their colleagues. It was so simple, I thought I had missed something.

I decided to write down the process I followed, not as some-sort-of pinnacle of best practices (far from it, I’m sure). But to hopefully share some advice, tools and steps that could help other developers.

The brief:

Create a web application where employees can receive “badges”, i.e. recognition for completing learning activities. Other websites offer similar functionality, like youracclaim.com and support the Open-Badging Interface, so the app should provide portable badges.

I speak with a few vendors first to see if we can purchase this. The learning system we use is pretty new (Pathgather) but it has an awesome REST API and I’ve already written the Python wrapper for it. If we used a third party solution I deem that we’d need to build all the custom integration anyway, and I reckon I could fill the remaining gaps to build it all inhouse.

Architecture:

There’ll be potentially 30,000 users (how many employees we have) who are spread across 50 countries. Traffic will be low, so simplicity of administration is key. We don’t need microservices, Docker, etc. etc. This is just a dumb and simple web application.

Day 1

Hour 0 — researching alternatives

I always try and land somewhere in-between “not invented here” and “never invented here”. So first, I scan the internet for Django applications for badging. I find 2, one which is terribly old and abandoned, with issues like “Does this even work?” from 3 years ago. Then, another, which looks more promising. After reading the source of the promising option I find it to be a lot of boilerplate and no actual substance, it wouldn’t save me any time.

Hour 1 — setup Django

All projects start with git init — check-in early and often. I add *.pyc and __pycache__/ to .gitignore

Setting up Django is always relatively straightforward.

Create a virtual environment, in my case I go for Python 3.6. virtualenv env --python=python3.6
Within my activated VirtualEnvironment I pip install django which gives me Django 2.0.3
Create a Project django-admin startproject dd_badger and then straightaway create an application in the project python manage.py startapp core . Only project settings and WSGI will live in the project directory.

Hour 2-3— setup authentication

Before I do anything with a corporate web app I always establish how users are going to logon. You need to solve this problem early as it typically has so many knock-on effects, like when you’re coding permissions, access, or identity.

Any internal corporate application needs Single-Sign-On. This generally means that users who are employees don’t need to register.

The form that takes in most Enterprises (95% of whom use Active Directory), is normally either ADFS (Active Directories Federation Server) or a 3rd party like Ping Federate. The protocols used for federation (OAuth, WS-Federation, SAML), whilst interesting, are often irrelevant because all the detail lies in the implementation of the identity provider and nobody follows standards.

Setting up ADFS as an authentication provider is no picnic, but we also have Yammer. Yammer comes with an OAuth2 endpoint, and someone has already made a Django Authentication Provider for it.

I configure 2 Yammer applications, one for localhost for development, and one for production you can log in and automatically register any employee.

My next step is to configure Django’s Social Auth plugin. Which is installed by pip install social-auth-app-django . I now have 2 requirements, so I set the requirements of requirements.txt so that I won’t forget in production.

The Social Auth app depends on a common package for Social Logins, which I recommend you checkout. I’ve gone through the steps to configure it before, so it’s pretty simple

Add social_django to INSTALLED_APPS
Add social_core.backends.yammer.YammerOAuth2 to AUTHENTICATION_BACKENDS
Set LOGIN_REDIRECT_URL = ‘/badges/’
Set LOGIN_URL = ‘/login/yammer’
And finally, set SOCIAL_AUTH_URL_NAMESPACE = ‘social’ so I can refer to the social links inside view templates.

Lastly, I put the secure keys into a separate file. Django imports the settings submodule. Even though settings.py comes as a file by default, I create a directory called settings and a file __init__.py then move all of the existing settings into that file. Now, in __init__.pyI from .secrets import * and create a settings/secrets.py file. Before I forget, I add that file to .gitignore . Why? Because I want to store both local and production connection secrets in there for Yammer and our Learning API.

Once I check that in to Git, it’s there forever. So if I open up the repository to 3rd party contributions in the future, someone could trawl through git history and find it.

Lastly, I make a note in README.md with an example of what that file should include. Just in case I step out in front of any buses the day after.

Once the Yammer setup is done. I need to follow the Django database migration process and restart the service.

I create a “hello world” view with @loginrequired that then forwards me to Yammer’s login page and onto our ADFS endpoint. Once all this is done, because my superuser account has the same email address as my work Active Directory UPN. It automatically links the 2 accounts like this:

In future, any other users will just log in and it will create the User record first and then the linkage.

I can log in using my corporate credentials (with 2FA) and no registration required. Employees should find this intuitive.

Hour 4 — Create some models

Databases are fun.. I setup 2 models in Django’s ORM. One for the award, and one for the fact it has been awarded (and when).

Lastly, I implement __str__ on both models because it’s just horrible to debug and/or administer instances if you don’t know what they’re for.

I use UUIDs as the primary keys, with the default value coming from Python’s standard library module.

I try and avoid premature abstraction when it comes to DB design. If a scenario crops up that means that awards are not coupled to our existing learning system, then I can change it. Until then, it stays simple.

Hour 5— Create a base template

Working for a (very) large company, we have our own internal Web UX design portal. It contains all the includes, CDN links, and rules around UI design to ensure any internal portals are consistent.

It is based on bootstrap 4, so I start by using one of the templates and turning it into a template called base.html and store it in the templates directory in the core app. The base template has the header, a menu, a message in the top right saying your login name (which is fetched from the social auth module).

The contents of the base template are set through the block content attribute in Django. So all I need in a view template is

{% extends “base.html” %}{% block content %}Hello world!{% endblock %}

then it will render a nice page will all my headers and footers.

**Yes, my corporate name has a snake in it (remember that for later)**

Hour 6 — create basic views

I then created 4 views,

The home page to see a list of badges you had been awarded as well as those you could potentially recieve
A detail page for a badge that you had not received, to show more information about it
A verify page that checks a user (by email) has achieved a certain badge
A page to design and create a new badge

I’m a fan of the MVVM (Model, View, View-Model) pattern, whose principals are to try and avoid throwing your entire database at the View (in Django’s case, the template) just incase you might need a particular field. In a nutshell, you give the view crafted data to minimise the amount of logic you need to code inside the template.

In practice, this view (e.g. 3) gets the badge, searches for the path in client, which is a lazily-instantiated instance of the Pathgather API client. It then gets records of someone with this email having done the learning material and shows their percentage complete.

The design-a-badge page I use Django’s builtin Form model and use a URLValidator on the image_url field and a custom validator on the slug field:

RegexValidator(‘^[0–9a-z-]*$’, ‘Only lowercase letters and hyphens are allowed.’)

I then install django-bootstrap4 a package for providing tags to render Bootstrap-style forms correctly and use the bootstrap_form with a bootstrap_messages . Not forgetting, of course to add that to requirements.txt

Now, testing that out, it validates the slug field and the image URL field and provides both errors at the top, as well as next to the fields.

Hour 7 poke around

I spend the final hour poking around in the UI, thinking about the UX and testing things like having no badges, lots of badges, giving fake ID’s.

5pm I finish. That’s it for today. Wait — I’m not staying up til 2am furiously keyboard mashing? Nope. Good work always comes after a good night’s sleep and I have plenty of other things to be doing. Like reading the kids some stories.

Day 2

My daughter left her school shoes on the beach, so I spent the first hour of the day looking for them. Thankfully, someone had found them and placed them by the cafe. So I dropped them off at school and started work.

Day 2 is deployment. No, the app is nowhere near finished yet. But I need to establish what the production environment will look like as early as possible and iron out the kinks and develop a process for deploying.

Hour 0–1 Azure Web Apps

I decide to start with a Web App template in Azure, I follow the tutorials and eventually establish that it doesn’t match what I’m trying to do.

I have a little moan about it on Twitter and someone from Microsoft team pings me asking what went wrong. I explained a little and then Dan Taylor from the Azure team reaches out to me and sets up a 1 hour Skype call (this is the following week). I go through on Skype what I did and what I thought they needed to improve upon.

Update: Microsoft have since fixed these issues, you can read more in my other article https://tonybaloney.github.io/posts/django-on-azure-beyond-hello-world.html

Hour 2–4 Compute instance configuration

Instead of going further down the rabbit-hole I back out and go with what I know, configuring a Linux VM. The kind folks at Bitnami have a pre-configured Ubuntu 16 (ie. Python 3) compute template in Azure with Django installed and MySQL! Win!

Whilst configuring the instance, it prompts me for a MySQL password and an SSH password. I generate 2 in LastPass and store them in my vault for later.

Upon logging in to my new VM, I go to the apps folder and start to pull a copy of the application. Both GitHub and GitLab have “deploy keys” which are special SSH-keys for deployment only. The benefit is you can set granular permissions, such as read-only. This is normally my preferred approach to early-deployment scenarios whilst things are still in test. Keeping it read-only means that any potential compromise of the server only has access to that 1 project and to read it, not to write to any of my Git repos.

I clone a copy of the project onto the server and start to configure the Apache HTTPD Virtual Host and Directory settings.

Point the WSGIScriptAlias to my wsgi.py path
Set WSGIDaemonProcess for 2 processes and 15 threads

Next I connect to MySQL locally, create the database and then update settings/secrets.py with the connection details to the database. After this I can run python manage.py migrate and it will publish all the tables on MySQL for me.

I then register another Yammer application for the new domain name that Azure allocated for me. I put the Yammer details into settings/secrets.py

Lastly, I need a valid SSL certificate. I follow the steps specific to this Bitnami template and we are rocking a nice Green padlock in Chrome.

Hour 5 — Fixing Emojis

Now, after setting all that up I try and login for the first time. Then I essentially get an error like this:

Why? Well, in the screenshot up the top I was testing locally and it worked fine. Now, I test with a MySQL database and it doesn’t work. The issue was that because my corporate name has a 🐍 in it and MySQL, by default doesn’t support multi-byte Unicode values, it blows up. This demonstrates why:

You should put emojis in tests,
You should test your deployment environment early.

Anyway, I’m not the only one to hit this issue, so after some Googling, I find an almost-accurate blog post and fix the database. To be honest, I might drop MySQL and switch it out with Postgres, because this is BS.

I fix it up and login works as expected, so I click around and create a couple of badges to test on. All works smoothly.

Hour 6 — spit and polish

Now that I have a server with the application deployed, I spend the next hour improving the interface:

Friendly error messages when it can’t find values in the database
Humanizing date values on the view using Arrow’s humanize
Testing various screen sizings and scenarios
Checking “long names” and how they render

I add a “Share” button when you have a verified badge that posts onto Yammer your award with a hard-link to the verification landing page.

Hour 7 — User testing

I’m ready for my first users, so I invite the team to login and try and create a badge and check the status of one’s which they may have already qualified for.

Actual names replaced for privacy reasons

Feedback is good — nobody asked for help whilst testing. The team discuss the implications and how they think users will react when we launch this. Everyone is excited, time to switch off for the weekend.

We had some friends arriving at 5.30pm and it was a beautiful evening with lots of wine and the BBQ.

The following Monday…

One of the key bits of feedback from user testing was that employee’s will see the badges as being more significant if they can demonstrate them to people outside the company. Even if they do leave, it makes sense for them to keep some form of recognition. We all agree, and someone points me to an Open-Badging Specification called OBI.

Demonstrating the importance of getting external feedback early, we add 2 scenarios to the application:

Ability to import a badge from youracclaim.com (like a Microsoft or Cisco certification)
Ability to export an internally-issued badge to the Mozilla Backpack.

Both are pretty straightforward, so after an hour on the train into the office I get the import working and on the train home I get the export working.

The OBI code is quite fragile, so I write up a series of unit tests for it using pytest-django.