Continuous integration for small teams

Here at Imerso, we want to test and deploy continuously while we also try to maintain some level of stability in our production environment. We’ve focused on having low operations overhead in our daily work, and we’ve tried to use readily available mainstream solutions instead of putting too much effort into configuring our setup. We use Circle CI to build, test and deploy our code, and our code is hosted on GitHub. This post explains how we’ve pragmatically set up a continuous integration / continuous delivery pipeline (CI/CD), and we’ll explain how we deal with hotfixes to production and DB migrations.

Production and staging
We have two environments continuously running our entire stack, stagingand production. production is the live environment for our users, stagingis used internally for testing and presenting the very latest features before we promote them to production. Having an extra staging environment does create some overhead, since we need to duplicate everything. But we found this worthwhile, even in very early product development stages. We map our git repositories to the environments in this way:

• master branch is deployed tostaging
• git tags like v1.2.3 are deployed to production

This way, in our day-to-day we only interact with git and GitHub, and let the deployments be handled by Circle CI. Our circle.yml looks something like this:

(...)
deployment:
  prod:
    tag: /v[0-9]+(\.[0-9]+)*/
    commands:
      - make deploy-production
  staging:
    branch: master
    commands:
      - make deploy-staging

This way, when merging a branch into master, it goes directly to staging. When pushing a new release tag, that version goes directly to production:

git tag -a -m "New prod release" v1.2.3 <commit>
git push origin v1.2.3

Multiple repositories, not one monorepo

Many a blogpost has been written on the monorepo vs multi-repo topic, and google and Facebook are famous for having gigantic monorepos with thousands of developers contributing. We believe there are good arguments for monorepo scaling better, and with a certain team size or product complexity that probably makes sense. But for us it comes down to using multiple repositories because of the much simpler CI/CD setup. We currently have three repositories:

• Android App, Kotlin/C++ deployed to Play Store
• Webapp, Scala/JavaScript deployed to Google Kubernetes Engine (GKE)
• 3D mesh processing microservices, python/C++ deployed to GKE

Even though our product in a loose sense consists of everything in that list, they are three very different codebases. Different programming languages, different test systems, different deployment targets, i.e. entirely different CI/CD configuration. At some point, maybe we will put the effort into merging them all into a monorepo, but we think the effort of configuring partial builds, partial deployments etc is just not worth it as of now.

git branching workflow

As is usual nowadays, we use a feature branch workflow with git, where we can discuss features and do code reviews on feature branches before they are merged in. This is a breeze with GitHub Pull Requests with its relatively recent review features. We would love if GitHub Pull Requests had better support for multiple branch revisions¹, but lets not digress. Feature branches are typically tested locally while being developed, but we also run all tests including lint checking in our Circle CI builds.

When we merge a Pull Request, the Circle CI deployment will put it on staging. From time to time, we do find it useful to deploy a feature branch to staging before merging it to master. This will obviously not scale if multiple developers try to control staging at the same time, but for the time being that works fine and we don’t see the need for configuring a per-feature-branch deployment setup.

We try to keep “production quality” (™) on what we merge into master and generally promote to production often, but sometimes we want to do some ad-hoc QA testing first, verifying stuff still works. We usually deploy to production every 1–3 working days, typically leaping a few feature-branches at a time.

Hotfixes to production

Since we use a single mainline master instead of a more involved branching model for releases and development, how do we get hotfixes out to production? Typically, we can just merge the hotfix to master and create a tagged release for production, but sometimes there are undeployed changes on master (in git log v1.2.3..master) that we don’t want to deploy just yet. We could revert the undeployed commits, apply the hotfix and create a tagged release, then re-apply the undeployed commits. But, we find the best solution is to create a temporary hotfix branch which is also merged into master:

git checkout -b burning-house-hotfix v1.2.3
# ... do hotfix and commit it
git tag -a -m "Hotfix release" v1.2.4 HEAD
git push origin v1.2.4 # deploy to production
git checkout master
git merge --no-ff burning-house-bugfix
git push origin master # deploy to staging

Rollbacks and DB migrations

If some unforeseen problem occurs in production after a deployment we want to roll quickly back to the previous good state while investigating the problem. For our GKE based web services, we can roll back in a matter of seconds with kubectl rollout undo. This is all well and blissful until we have to consider DB migrations too. If the last deployment also migrated the DB schema (say, added a new column to a table) a rollback would use old code with the new schema which would probably lead to No Good.

First, to back off, we do DB migrations as part of the Circle CI deployment using Flyway (through sbt in our case). The circle.yml deployment specification for our webapp actually looks more like this:

(...)
deployment:
  prod:
    tag: /v[0-9]+(\.[0-9]+)*/
    commands:
      - ./cloud_sql_proxy -instances=$PROJECT:europe-west1:$PROD_DATABASE=tcp:3400:
          background: true
      - sbt flywayMigrate:
          environment:
            SQL_DB_URL: jdbc:mysql://localhost:3400/imerso
            SQL_DB_USER: imerso-dbadmin
            SQL_DB_PASSWORD: ${CLOUD_SQL_IMERSODBADMIN_PW}
      - make deploy-production
  staging:
    branch: master
    commands:
      - ./cloud_sql_proxy -instances=$PROJECT:europe-west1:$STAGING_DATABASE=tcp:3400:
          background: true
      - sbt flywayMigrate:
          environment:
            SQL_DB_URL: jdbc:mysql://localhost:3400/imerso
            SQL_DB_USER: imerso-dbadmin
            SQL_DB_PASSWORD: ${CLOUD_SQL_IMERSODBADMIN_PW_STAGING}
      - make deploy-staging

To mitigate the problem with DB migrations and rollbacks, inspired by this very good writeup on the subject, we have created a best practice convention when adding features involving DB migrations, essentially:

• Current up-to-date good state, v
• Release DB migration only, v+1
• Release new feature, v+2

By following this practice, we can safely roll back to v+1 if our new feature introduced problems.

In more laid out terms, that leads to these steps:

1. After feature is reviewed and ready for merging, split out the commit(s) involving the DB migration and if necessary the absolute minimum code changes necessary to support the new schema.
2. If DB migration is not super trivial, test it on a relevant (populated) DB, preferably on a clone of production DB.
3. Merge DB migration (i.e. v+1 deploy to staging)
4. Merge new feature (i.e. v+2 deploy to staging)
5. Repeat steps 3 and 4 for production

Rolling back from v+1 to v will typically not be OK here, but so far we haven’t had any issues with the v+1 release (knock on wood). If we wanted to be even more safe, we could create and deploy an intermediate code state between v and v+1 that is forward compatible with the upcoming new DB schema. For now we don’t see the need for always doing that, which means we cannot roll back past a DB migration v+1 release, but we can roll back anything else in between. It’s a pragmatic priority we’ve made, we choose to put more emphasis on feature velocity.

Conclusion

Hope you found some of this interesting, I’ve tried to present how we’ve laid out our CI/CD and how it ties up with our everyday workflow. Also, why we chose multiple git repositories over a monorepo, how we interact with two deployment targets, and how we prepare for rollbacks when the house is burning.

Thanks for reading through! Drop us a line if you have questions or if you think we’re doing it wrong.

[1] On GitHub, a Pull Request is tied to one specific branch and the most straightforward way to update a branch (e.g. after a review) is to add fixup commits on top of the existing branch. Since we want to keep git history clean we prefer squashing in the changes and updating the branch. If we force push (git push -f)the updated branch, there is no trivial way to see the “second order diff” between the old and new branch revision. Also, while commenting inline on code changes is straightforward, there isn’t really a good way of commenting on commit messages.