Basic (Continuous Integration?) Deployment with Ansible, Docker, Jenkins and Git

Recently, I tried to build a Continuous Integration (CI) deploy flow for my team. Then I start this study alone, and come up with these thoughts.

In the end, as we still didn’t finish all the test cases and we decided to skip it at this moment. It will be shameful if we call this deployment a CI flow. ANYWAY.

This article mainly provides the concept of the architecture. I won’t talk too much on how the precise steps being implemented.

Objectives:

  1. Jenkins manages WHEN and WHAT to do.
  2. Ansible manages HOW and WHERE to do.
  3. Docker ISOLATE your application with the host, if the host can run docker, you can deploy your application.

What scenarios will be achieved:

  1. Your teammates can help doing the deployment simply by clicking buttons using Jenkins.
  2. Everything is scripted. Tasks will be repeatable and debug-able.
  3. Deployment script will be put in the application directory and managed by git at the same time. This is a benefit if you are in a same time and doing application and system management at the same time.

4. of course there are much more …

Four main tools:

http://docs.ansible.com/ansible/intro.html#

To me, Ansible is the best tools for managing my deployment code, because:

You write commands to set the hosts’ states instead of running an action.

No more wasted time on writing if-else cases to check a file’s existence and delete it. The only thing you need to do is just state it as absent.

file: path="/etc/some_directory" state=absent

details: http://docs.ansible.com/ansible/file_module.html

You manage your hosts by grouping and defining roles

You can describe 10.100.222.1–10 are api-hosts. describe api-hosts need ~/.ssh/deploy.pem to ssh for deployment.

You can describe 10.100.222.11–13 are mongodb-replica-set. and 10.100.222.13 will be mongodb-backup-source.

These kind of grouping and making roles help you to seperate HOW and WHERE to do the task very well.

Docker is a tool that I love and hate it at the same time. The good things are:

Isolate the environment setup from the actual host machine by providing you a VM-like Docker container.

No matter what host machines your infra team gives you, you can deploy to these machines provided that they run Docker. Also, you can control what OS your application running in whatever the host machines are actually running

Deploy multiple applications on the same hosts thanks to the Docker containers.

It is always painful when you deploy multiple application on the same host machine. Image your have two sails projects, one is running nodejs:5.9.2 and the other is running nodejs: 6.0.0. It is will be a trouble the you need to config the node version for the two projects. Fortunately, with Docker container, you just need to specify the node version and installation in the separate Dockerfile. And the nodejs will be installed in separate containers. The host machine will just need to run the two containers and things are done.

Besides, for some macro service architecture, it will be much easier to setup if you need to run multiple services on the same host.

Large community provides different kinds of basic Dockerfile

There are lots of basic Dockerfile that you can start with writing your own, like nodejs and rails. Even you can just start a Jenkins server by using

docker run jenkins

The Docker engine will pull the required DockerImage and run the Jenkins.

However, there are down sides of using Docker. Yet, to be honest, when you are an experienced Docker user, you will solve these problems easily.

A large DockerImage will result in a slow export, import and shipping DockerImage among hosts.

The first thing you SHOULD KNOW about writing Dockerfile is how the DockerImage is actaully built from the Dockerfile you wrote.

One of the terrible thing is like this, in simple words, you write you Dockerfile command by command and each command will create a layer to record the actual system change by the command. For example, your first command delete 50MB files from the system. And then your second command add 30MB file to the system. The layers will treat it as a 80MB file change in total.

Of course, there are best practices to avoid this senarios.

Dockerfile should be well written in sequences to make well use of caches

Everytime you build a DockerImage, the Docker engine will cache the layers so that the building process will be faster next time.

One the example is that, for a nodejs project, it is always needed to install packages and also copy the application code into the DockerImage. These two actions both changes the result of the DockerImage and affect the caches. However, comparing the frequency of these two actions, you will find that running the installation of packages will affect the caches less.

A well-known tools manage tasks for servers. Actually, not much I can talk about Jenkins. It just easy to use and powerful.

Powerful plugins for hooking up the Jenkins and Github

For git users, it is a common task for you to hook up Jenkins and Github for deployment. Once you code changes in Github, your Jenkins get notified. Also, this kind of hooking is an essential parts of a Continuous Integration System.

GitHub plugin

This one is what I am using.

Easy to setup slaves to scale your tasks in Jenkins

There are lots of cases that you need to create slaves for your Jenkins. for example,

  1. There are lots of tasks to do. And you need more workers
  2. There are some tasks require certain environment.

The first one is easy to understand. For the second one, for example, we need to deploy a DockerImage to some hosts when Jenkins noticed code is changed in Github. Where to pull the code? Where to build the DockerImage? In this case, you will need a Jenkins slave with Docker installed (it’s actually just a ubuntu installed Docker) to handle all the Docker related stuff.

( You can say that why not just install Docker in the Jenkins host. The main design concepts to me are:

ALWAYS MINIMISE THE NUMBER OF KINDS OF BUILDING UNIT IN YOUR SYSTEM.

DON’T MAKE YOUR BUILDING UNIT COMPLICATED.)

The plugin that help me a lot when setting up slaves is this:

SSH Slaves Plugin

Git

If you are already using Ansible, Docker and Jenkins, I am sure you know what is Git and what it helps.

Two Dimensions Deployment:

I try to separate deployment flow, in concept, into two dimensions, vertical and horizontal.

The vertical one focuses on how code change become a result of an application update in a server.

This dimension is solved mainly by:

  1. the hook between Jenkins and Git
  2. the Docker and Ansible slaves setup
  3. Build, export, ship and import the DockerImage from Jenkins’ slave to the host by Ansible script.
The horizontal one focuses on how a group of hosts are updated.

This dimension is solved mainly by:

  1. Jenkins tasks start an Ansible script.
  2. Ansible ship and import the images in parallel
  3. Ansible restart the Docker containers in servers one by one to minimise down time.

So Now…

Let’s try to talk about some real-life cases:

Assuming we are going to have API servers and Web servers and we are going to have a auto-deploying system (or Continuous Integration) in staging and dev and a click-to-deploy flow in production.

Our building units will be:

  1. Docker Instances

Yeah, I can’t list more. Let see how we build with only Docker Instances.

Jenkins

  1. Just run this in the Docker instance
docker run jenkins

2. Install plugins, hook GitHub and build slaves (maybe talk about it in another article)

Jenkins Slaves

  1. With Docker installed, you just need to install Ansible in this instance.
$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:ansible/ansible
$ sudo apt-get update
$ sudo apt-get install ansible

API Server and Web Server

  1. With Docker installed, nothing more you should do. Just leave the jobs to Ansible script to start a Docker container.

MongoDB

  1. Just run this in the Docker instance
docker run mongo

(of course, you need to write a config file for replica-set, and mount volumes to store the data out of the Docker Container, but these can be scripted)

Redis

  1. Just run this in the Docker instance
docker run redis

(of course, you need to write a config file for replica-set, and mount volumes to store the data out of the Docker Container, but these can be scripted)

The Evil Thing
If you are clear about this building concept, you will release that, except Jenkins requiring some unavoidable manual work:
1. every instance only need some standard command to do the extra setup and start its service.
2. With standard commands, things can be scripted.
3. With scripts, this architecture can be reproduced easily.
Also, as every basic building unit is a Docker instance, you can write a script to do it!
So everytime you need to scale out, you receive a new host / create a new host on cloud. Then you run script to install docker on it, add the IP to the correct inventories. DEPLOY!

In Jenkins

Jenkins only control WHEN and WHAT to do

We can create different cron job and task in Jenkins with only simple command to start the Ansible job.

(As there are always some variables when running the Ansible commands, I usually wrap it with a shell script stored in the code base)

ansible/run.sh

In the Code Base:

Ansible control HOW and WHERE to do

For Ansible script, I mainly follow the official best practice.

LINK

  1. You can see we separate variables and hosts IPs and groups according to different environments.
  2. For roles, I usually name them with the tasks’ names. It seems more flexible. (any suggestions for this?)
  3. The ***.yml at the bottom are the main playbooks to be run for different tasks.
  4. The ***.sh is the wrapper for calling the Ansible command in order to simplify the script in Jenkins.

For Docker, you can see we have Dockerfile, Dockerfile_ecg. This how we try to handle different environments if needed.

We use Ansible to replace the Dockerfile with a suitable one before we build the DockerImage.

(If you still remember, what the Dockerfile build should be independent of the host machine. But for our cases, since one of the environment needs proxy to do connection and one of the cloud is going to be deprecated, that’s why)

Summary

I know this article may be hard to understand since I just talk about concept but not the whole implementation like other tutorials outside. Yet, to me, having a deep thought on how architecture should be made is the most interesting thing in engineering. Concepts also give you a structure to consolidate your knowledge and build your own stuff. Also, it will be a lot to talk about if we write these in step by step.

Feel free to give suggestions or ask questions.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.