Micro-service deployment process using Jenkins, CloudFormation, ECS and Docker

Last week I wrote about how we had created a Slackbot servant to do our bidding called JARVIS. This week, I’d like to share a process we automated to deploy micro services in a repeatable fashion. We call it CruiseControl and it glues together a variety of tools in a way that we’ve been able to re-use for many different cases.

What does it do?

CruiseControl allows you to focus on developing the code for your service rather than having to worry too much about deploying it in a repeatable, reliable way. It revolves around creating a service in a Docker container (or linked containers) that can then be deployed to Amazon’s EC2 Container service. It handles load balancing, service discovery (via DNS), monitoring and, via Jenkins, an easy way to promote or release the service in a repeatable way.

One of the main philosophies our small DevOps team (yes, I know — DevOps is a culture!) has been trying to put forward as much as possible is making everything as standardized and repeatable as possible. Deploying a snowflake each time is really painful — but having a good pattern makes deploying new releases almost a non-event.

What does this all look like?

We’ve used Cloudformation to do almost all of the heavy lifting. Using a common Cloudformation template, we create the ECS task definition, ECS service, ELB, Route53 DNS entry and (via a separate template), Cloudwatch alarms for CPU and memory service utilization. Everything is controlled by a Jenkins job which builds the Docker image, pushes it to a registry (we use Dockerhub) and prepares some deployment rules YAML files. Then, using the Jenkins “Promoted Builds” plugin, we invoke the scripts that talk to Cloudformation to do the heavy lifting with ECS.

Where can I find this fantastic toolset?

We’re building our github repo out with new tools every week it seems. CruiseControl is available at https://github.com/Signiant/CruiseControl