Coupling SaltStack and Docker

Years ago, before SaltStack, I was using fabric. I had a fabfile that would zip up my code and copy it over to my server. Deployment was simple… for the first few updates. Then came the day when the micro instance started failing. At the time I was using AWS and I spun up a larger instance. The selling point to cloud providers is that you can “easily scale” your application. The fine print to that statement is, “only if you architected your deployment correctly”. I spent days comparing all the hand installed packages and their versions on the failing micro instance to the new instance. Once the new server was finally running my code error free, I vowed to find (or if need be, develop) a better solution. It should be as simple as a fabfile, but capable of managing packages and versions. The ideal solution would let me run a single command and magically propagate code to a new server, or many servers, while also installing the required packages to run the code.

SaltStack

After a few days of research, SaltStack was the obvious choice. Well supported and documented by a massive open source community. It features simple YAML config files known as Salt State files and the backend was written in Python (same language as my codebase). I created Salt State files for supervisor, nginx and my application code. The best part was that I could do a simple:

salt 'tornado-web-*' state.highstate

and all of my production servers (new or old, with a name starting with “tornado-web”) would be updated and running identical configurations specified in the Salt State files. This solved all my issues I had with the fabfile, but it created a new set of problems. First, removing a Salt State file, doesn’t remove the corresponding package from the server. Example, let’s say you were initially using Apache to run an HTTP server and then decided to switch to Nginx. You would need to have a new Salt State to purge the old Apache packages and a new state to create the Nginx packages. Another, much bigger nightmare with Salt was dealing with the application development environments. Example: I wrote a new feature that requires a new package to be installed on the servers. Updating the Salt State files to reflect the new code was laborious. If I did remember to properly update the Salt State file, the team often ran into errors when they eventually pulled the new code into their development environment. In the end, a simple new feature that required a new package caused all sorts of headaches.

Docker

When I first heard about Docker, I didn’t see what the hype was all about. I dreaded using virtual machines and resorted to python’s virtualenv. A python virtual environment is very straightforward for development. You only need to pull code and not a gargantuan, multi-gigabyte, virtual machine. Managing the versions of all your virtual machines is another nightmare. After many friends explained the technical details of how a Docker container is different than a virtual machine (great explanation on Stack Overflow) I decided to try it out at a hackathon. Ignoring installation, I had written and built a Dockerfile that installed everything required to run a web application (Supervisor, Nginx, Python3, et cetera) in less than an hour.

To build the Dockerfile, make sure you’re in the same directory as the Dockerfile and run (don’t forget the trailing period):

docker build -t tornado-web .

To tag the image, we first need to get the image id:

docker images

To tag the image, it’s:

docker tag <image_id> <hub_account>/<image_name>:<tag>

An example would be:

docker tag 7d9495d03763 myhubaccount/tornado-web:latest

Finally I pushed it to the Docker hub so the team could run it:

docker push myhubaccount/tornado-web

Installing the Docker Engine daemon on an Ubuntu server was less ideal, but still pretty straightforward. Once the Docker Engine was running, it was magical. I was able to start a container running my web server with ONE line:

docker run myhubaccount/torndao-web

Members on the team were also able to execute that “docker run” command on their laptops to run the server locally. Best part, I could scale this application by spinning up more instances, installing docker engine and executing that “docker run” command. Infinity better than the original fabfile, but adding or updating servers would require me to manually ssh into each server to run the latest docker image. Needless to say, I bought into the Docker hype and use Docker containers for everything now.

SaltStack + Docker

Midway through installing the Docker Engine on the Ubuntu server I realized I should have used SaltStack and created a Salt State file.

Pro Tip: If you’re ever doing something that you don’t ever want to do again, but inevitably will, write a script!

Since Medium doesn’t play nice with multiple files in a Github gist, I will refer to the gists by name. There is only one new file required for Salt to install the Docker Engine and that is the docker/init.sls, gist: SaltStack-Docker-Init.sls file. In short this does everything on the Docker Ubuntu installation page, purge packages, installs dependencies, et cetera. A critical part is the last state, “docker-service”. This will make sure the Docker Engine is always running.

The platform/init.sls, gist: SaltStack-Platform-Init.sls file should already exist if you’re using Salt. This file typically pulls your code from Github, or acquires it by other means (maybe you copy your code directly from the build machine). We’re just going to append two docker states to make sure we’re running the latest docker image available from the docker hub. To keep our docker image portable (ability to use it on our development laptops AND production servers) we bind our code directory from the host file system to a docker directory. This enables the developers to use PyCharm or WingIDE locally and enables us to do a simple git pull on the production server’s file system without having to jump into the running docker container (win, win).

For developers, you can bind to a local directory and port in the docker run command:

docker run -d -p 80:80 --name webapp -v /Users/me/Documents/webapp:/home/www/tornado -it myhubaccount/tornado-web:latest

The last file is should also exist if you’re already using Salt. This is the top.sls file that tells Salt which States to run on groups of servers. In this instance we’re telling Salt we want our docker and platform states to run for all tornado-web-* named servers.

On our Salt-Master we can easily run:

salt 'tornado-web-*' state.highstate

and our servers will now be our Tornado web server running in a docker container.