From commit to production

Every development team has a development and deployment workflow.
This workflow could be completely manual, where every step is done in a traditionally way with hand-made code merging, testing and validation, scheduled production deployment, and perhaps the approval of several bosses before any new functionality or hotfix goes public. In this post, however, we will cover our automated workflow.

There is no silver bullet, as every workflow must fulfill the specific
needs of a project. Each project may have its own workflow; it’s not always one size fits all.
While every squad owns its workflow with slight variations, they all share the same principles:

• Simple. It must be as simple as possible, no complications, no bureaucracy.
• Fast. We need to go live in production with new changes in minutes, not hours.
• Automated. The less human intervention, the better.

Our continuous integration and deployment stack consists of:

• GitHub. All the source code is stored under DVCS and we are using Git+Github for it. GitOps FTW
• CircleCI. Our Continuous integration platform. Docker based, as a service, and fully managed. Configuration is stored in the same Git repository.
• Kubernetes. All our services are deployed as containers(Docker+Helm) to managed Kubernetes clusters.
• Slack. We use slack for communicating within the team, and also for notifications and for ChatOps.

So, how do we put all this together? How does a developer go from committing a line of code to deploying in production?

First of all, some clarifications about the environments:

We work with several environments. We distinguish between production environments and non-production environments (sorry that is obvious!). The non-production environments are used to test and validate production readiness. We define these development environments as such:

• Feature or hotfix environment. Isolated environment used to validate a single feature, hotfix or user story.
• Integration or master environment. All the features are merged and tested in the integration environment from a developer point of view.
• Staging environment. Selected snapshots from the master environment are pushed to the staging environment, and business validation decides whether it is ready for production.

In order to understand how we deploy, it is important to understand the distinct team roles.

• Product Owner (x1)
• Scrum Master (x1)
• Technical Lead (x1)
• Front-end development (x5)
• Back-end development (x4)
• DevOps (x0.2, shared with other teams, yes!)

So there is no SCM engineer, no deployment engineer, and a DevOps engineer that splits his time amongst multiple teams. Our push to production does not rely exclusively on the time and expertise of a dedicated DevOps engineer. That would be a huge bottleneck and also irresponsible. The solution:

Anybody can make a production deployment at anytime

To ensure this can happen there are 2 principles that must be always present:

• Quality. To ensure quality we need to define the correct workflows and tools that minimize errors, bugs and regressions. Unit, integration and E2E testing are our friends, but we won’t talk about them in this post as they deserve a full story of their own in the future.
• Simplicity. Everybody is not a systems engineer, this means the deployment mechanism needs to be a pice of cake for everyone. Let’s talk about this.

How to make my commit go to production?

As a developer I want to develop a new feature. The first thing I do is to checkout a new branch:

git checkout -b feature/key-123-new-aswesome-feature

Where key-123 refers to the key issue in the issue management tool we are using (Atlassian Jira).

On the first push to Github, CircleCI will be triggered and will:

• Build the source code
• Run all the automated tests
• Generate a dynamic site in a separate namespace in Kubernetes to test the feature independently.
• Create a notification via Slack. The Development team will receive a URL where the code has been deployed to.

What happens when a git push is triggered

A single push commit turned into a full independent site where the feature functionality can be validated on its own.

We saw how the code can be deployed to one of 4 types of environments described above.

• F̶e̶a̶t̶u̶r̶e̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶
• Integration environment
• Staging environment
• Production environment

In order to deploy our code into the integration environment we need to create a pull request and merge it into the master branch.

Code review passed, LGTM!

When merging the pull request into the master branch it will trigger the same chain of events as before (feature branch), but in this case the code will be deployed into an integration environment in the Kubernetes cluster.

• F̶e̶a̶t̶u̶r̶e̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶
• I̶n̶t̶e̶g̶r̶a̶t̶i̶o̶n̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶
• Staging environment
• Production environment

Deploying into staging and production environment is slightly different, but also very easy and GitOps based. This means that with a single git operation a deployment will be made. The operation involved for this kind of deployments is git tag . As simple as it sounds.

Creating a tag that follows the regular expression:

^v[0–9]+(\.[0–9]+)*-rc\.([0–9])*$

and pushing it will trigger the CircleCI+Kubernetes+Slack workflow, but it will deployed to the staging environment. Example:

git checkout master && git pull origin master
git tag v1.0.0-rc.1
git push origin v1.0.0-rc.1

All done! We can repeat the process for rc.1 rc.2 … if we need to iterate and fix some bugs before releasing into production.

• F̶e̶a̶t̶u̶r̶e̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶
• I̶n̶t̶e̶g̶r̶a̶t̶i̶o̶n̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶
• S̶t̶a̶g̶i̶n̶g̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶
• Production environment

Whenever we are ready to deploy to production, anybody can create the corresponding git tag and push. It needs to match the following regex:

^v[0–9]+(\.[0–9]+)*$

For example:

git tag v1.0.0
git push origin v1.0.0

Voilà! Our code has been deployed to production!

• F̶e̶a̶t̶u̶r̶e̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶
• I̶n̶t̶e̶g̶r̶a̶t̶i̶o̶n̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶
• S̶t̶a̶g̶i̶n̶g̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶
• P̶r̶o̶d̶u̶c̶t̶i̶o̶n̶ ̶e̶n̶v̶i̶r̶o̶n̶m̶e̶n̶t̶

What we didn’t talk about in this post:

• Testing under the hood (E2E, Integration, and Unit tests).
• Infrastructure we are using and how is everything configured.
• Deployment scripts. This is where the magic occurs!

Is this the best deployment workflow?

Probably not, but it fulfills all our current needs.

Why not use Spinnaker or other tools

As we said, we wanted to keep it as simple as possible. We might consider other workflows or tools in the future when we have bigger needs, like A/B deployments, different versions, etc. We could also achieve this by hacking our own deployment scripts, but we will continue our research when the time comes.

If you want to know more, stay tuned for future readings!