Onfido Hack Day (pt. 2) — Kubernetes Unicorns

This is the second in a series of blog posts about the projects developed during the Onfido Hackathon earlier this month. If you missed part 1, definitely check it out here.

In this post we are going talk about a project that makes deploying, testing and experimenting with new features, changes or fixes super easy on Kubernetes, as close to the staging environment as possible.

Kubernetes Unicorns

Most of our services at Onfido run on top of Kubernetes, so it made sense to have at least one hack related to it. To get a high level overview of how we run our services and operate Kubernetes, check out a post I made earlier this year.

The inspiration for the project came from reading a blog post on how GitHub recently revamped their development environment to run on Kubernetes and how each branch can be deployed with one click. So we asked ourselves: ‘Why don’t we have this? One-click deploy of any branch would be a super cool feature to have at our disposal’. So we decided to build it.

The hackathon starts, we gather our team and we start brainstorming. We currently have automated builds and deployments for development and master branches, pushing to staging and production respectively. For new features, changes and fixes, developers commit to their own branches, build and test locally and when everything looks good, they open a pull request to merge into development.

Our initial incentive for the project was to make local service testing obsolete by giving any team the power to deploy and test any branch with one click and have an isolated environment created for them with a unique URL for their branch. As we thought about the different components of the project, we started realising that this could be so much more than trying to solve our complicated and unsynchronised local environment. This means that any new features, changes and fixes would be tested and run in an identical environment to staging (very hard to replicate locally), with access to the same environment variables and configurations. This could also improve the quality of new releases and we could finally get rid of the ‘works on my machine’ syndrome.

20 hours of coding later, we have the hack mostly completed, split into 5 components:

• API (Python using Flask + Gunicorn + Gevent)
• Deploy worker (Bash using Kubernetes CLI and AWS CLI)
• Terminal deploy script (Bash)
• Frontend (HTML/CSS/JS using Materialize for design, typeahead.js for search UI, jquery for HTTP requests and Jinja2 for templating)
• Branch search (Uses BitBucket API to search for branches in our git repositories)

The full workflow (using the frontend) in action:

Branch deploy to Kubernetes using the frontend

Just to note before we go into details, we’ve setup a service (website-demo) that serves our website statically. On another branch (feature/hack), we commited and pushed a change to the title and subtitle of the main page of the website.

Let’s break down the workflow:

1. We go to the ‘K8s Unicorns’ service frontend, do a search for our ‘feature/hack’ branch.
2. The search UI will do some requests to the K8s Unicorns API endpoint that has the BitBucket branch search functionality, get the response and render the results.
3. We click on the branch from the correct repository and click ‘Deploy’.
4. The frontend will do a POST request to the K8s Unicorns API endpoint that will spawn a deploy worker process.
5. The deploy worker will pull the code (if not cached), build and push the Docker container to the registry, apply secrets (if necessary), modify the k8s templates and apply them to a new k8s namespace, create a k8s ingress object (using Traefik controller) and create the unique AWS Route53 DNS entry. This process can take from a few seconds to a few minutes, depending on the size of the service being deployed.
6. The worker returns the unique URL to the API endpoint which forwards the response back to the frontend.
7. We visit the unique URL and see the changes we made on our branch.
8. Profit!

To simplify the workflow for our terminal loving developers, we’ve also created a deploy script that can be run from the git repository path, automatically finds the repository and branch names and calls the deploy API endpoint (we bypass steps 1–3 above).

Branch deploy to Kubernetes using the terminal

Check out part 3 of the series here.