You Don’t Need $1MM for a Distributed System

Setting up containers, load balancing, and service discovery on light hardware

But wait!

I hear you say.

I can’t really learn those skills without having a lot of hardware!
It’s a catch 22!

First, stop talking so loudly. You’re taking up more space than I am. Second, it’s simply not true.

Where We’re Going

My aim with this piece is to begin to show the possibilities that are available to you today. All of the below was done in a single afternoon, on a $20/month Digital Ocean instance (i.e. about $0.50 worth of computing time).

Making the Application

For our application, let’s do something simple: create an HTTP server that does some work lasting a few seconds. The first thing to do (after installing Docker) is to launch a container. I’m working on Ubuntu, so I’m using that as my base:

sudo docker pull ubuntu
sudo docker run -i -t ubuntu /bin/bash
apt-get install -y vim python python-dev python-distribute python-pip
pip install flask
root@dockbox:/home/ubuntu# docker inspect 6b0262f0fe5c
[...]
“IPAddress”: “172.17.0.13",
[...]
root@dockbox:/home/ubuntu# time curl 172.17.0.13:5000
63386263
real 0m5.948s
user 0m0.000s
sys 0m0.009s

Cranking Up the Containers

Let’s kick it up a notch and launch 20 more of these things. First, let’s commit our container to an image so we can launch more:

docker commit 6b0262f0fe5c prime:v1
root@dockbox:~# docker run -d prime:v1 python /root/work.py
root@dockbox:~# docker inspect 09a72800be9e | grep "IPAddress"
"IPAddress": "172.17.0.16",
root@dockbox:~# curl 172.17.0.16:5000
55098829
for i in $(seq 1 20); do docker run -d prime:v1 python /root/work.py; done

Statically Load Balancing the Containers

First, we’re going to need nginx on our host:

apt-get install nginx
for i in $(docker ps -q); do docker inspect $i | grep "IPAddress" | cut -d" " -f 10 | sed 's/[^0-9\.]*//g'; done
service nginx reload
If you didn’t know any better, you’d think it was just an every day number.

What now?

One of the most frequent problems you have to deal with in distributed systems is how to manage where traffic is sent. This sounds like a great use case for service discovery.

Running Consul in the Containers

If you aren’t familiar with Consul, or service discovery, check out this nice introduction. Otherwise, I’m assuming you have a cursory understanding. Let’s dive right in.

Setting Up the Server

I personally am using this ppa for convenience’s sake.

apt-add-repository ppa:bcandrea/consul
Github’s gist syntax highlighting seems pretty broken.
service stop consul
service start consul

Setting up the Agents

Let’s kill our running containers from before, and modify our base image to set up the consul agent. You can stop a container by simply calling:

docker stop container_id
docker run -i -t prime:v1
apt-get install software-properties-common curl
apt-add-repository ppa:bcandrea/consul
apt-get update
apt-get install consul
BIND=`ifconfig eth0 | grep "inet addr" | awk '{ print substr($2,6) }'`
/usr/bin/consul agent -config-dir="/etc/consul.d" -retry-join 172.17.42.1 -bind=$BIND
{
"service": {
"name": "web",
"tags": ["nginx"],
"port": 5000,
"check": {
"script": "curl localhost:5000 >/dev/null 2>&1",
"interval": "30s"
}
}
}
/root/start-consul.sh &
python /root/work.py
docker commit 18feadc515f7 prime:v2
docker run -d prime:v2 /bin/bash /root/run.sh
root@dockbox:~# consul members
Node Address Status Type Build Protocol
dockbox 172.17.42.1:8301 alive server 0.4.1 2
ea66b40276c3 172.17.0.131:8301 alive client 0.4.1 2
root@dockbox:~# for i in $(seq 1 19); do docker run -d prime:v2 /bin/bash /root/run.sh; done
root@dockbox:~# consul members | wc -l
21

Dynamic Load Balancing

Now let’s take the information we now have from service discovery and use it to automatically modify our nginx config. We’re going to use consul-template to do this.

Preparing the Nginx Template

Once you’ve followed the instructions for installing consul-template, create a template file for consul-template to work off of:

Starting Consul Template

To start consul template watching the file, run:

./consul-template -template "/etc/nginx/nginx.ctmpl:/etc/nginx/nginx.conf:service nginx reload"
* Reloading nginx configuration nginx                       [ OK ]
upstream primes {
server 172.17.0.152:5000;
server 172.17.0.154:5000;
server 172.17.0.155:5000;
server 172.17.0.163:5000;
server 172.17.0.152:5000;
server 172.17.0.161:5000;
server 172.17.0.156:5000;
server 172.17.0.158:5000;
server 172.17.0.168:5000;
server 172.17.0.167:5000;
server 172.17.0.157:5000;
}

Setting Up the Web UI

The web UI is essentially just a wrapper around all of the HTTP API endpoints, but is much less cumbersome for humans to use. If you’re following along (using the ppa I mentioned earlier), you can easily install the web-ui to get a little better insight into what’s going on:

apt-get install consul-web-ui
ssh root@dockbox -L 8500:localhost:8500 -N
*Sniff* they grow up so fast

Where to Go from Here

I hope you see the possibilities of what we’ve explored here. There are so many more things we could start to do on this box to build skills a bit more, such as:

  • Randomly take down containers while under a stress test and see how many requests fail. How can this number be reduced? What might be an acceptable compromise?
  • What is the maximum number of requests our system can handle at a time? Could we tweak some operating system settings to make this higher?
  • How could we tag and release different versions of our software (e.g. development, staging, and production), while having them all co-exist peacefully in their own containers?
  • One box can only go so far. How can we get multiple boxes running a set of containers and manage them easily?

Helpful Commands

Here are some commands I found myself running regularly throughout this process.

List all containers

docker ps -a

Launch 15 containers

for i in $(seq 1 15); do docker run -d prime:v6 /bin/bash /root/run.sh; done

Stop all running containers

for i in $(docker ps -q); do docker stop $i; done

Delete all containers

for i in $(docker ps -a -q); do docker rm $i; done

Remove a container after it’s stopped

docker run --rm [...]

Other Resources

Senior Software Engineer at reddit

Senior Software Engineer at reddit