Building our entire testing environment in one box using Docker

To start off with, I’m Rajneesh, an engineer at Hike Messenger. Today, hike is the fastest growing messaging app in India.

In this blog I will cover how we created a ‘Server in a Box’; a one-box sandbox environment using Docker that runs the entire hike setup consisting of 20+ services. Not only did this simplify life for developers, it has helped us move faster than ever before.

Problems We Were Trying to Solve

1. New Environment Setup

Our local development environment setup was a total nightmare. Hike architecture consists of various services; Redis, Mongodb, Mysql, Python, Java amongst others. Configuring all these services in terms of the right versions, right drivers, the order in which they should be brought up, was a error prone process and simply put, really tedious and time consuming. It would require almost a day or more if any platform specific issue surfaced. Every single developer environment inside hike was almost always different than the other even if you tried to list down each and every step (Ideally it’s not possible in real word!) which lead to issues like “It works on my laptop but don’t know why it’s not working on yours!”.

2. Code Testing

We have a central QA staging environment, where QA keeps running the tests and verifying the stage of the system. When the team was small, this doubled up as a dev environment as well. But as we started moving faster with more developers (which meant more experiments), maintaining the health of this environment became a big challenge. Any break in this setup would cost us hours and severely affect our productivity. We really needed a sandbox environment, where they could test their changes without affecting others’ productivity.

3. Resource Utilization

It would have a provisioning of at least 3–4 under utilized AWS instances (plus a lot of effort from dev-ops) for an isolated environment setup per developer or per team. A VM based approach would’ve led to resource wastage, given the duplication of system bin, library files and low CPU utilization. The list goes on….

The solution to all of these problems was to have a completely automated way of creating an isolated dev environment which could be created on demand. After evaluating multiple options, we finally chose Docker. Docker contained all the dependencies, the order in which services should start, the versions of packages to be pulled & built and would run on a single VM.

What is Docker?

Docker is a tool that makes it super easy to create, deploy, and run applications by using containers. Docker allows a developer to bring together an application with all of it’s parts(such as libraries and other dependencies) and ship it all out as one package.

By doing so, the developer can ensure that the application will run on any other Linux machine regardless of any customized settings.

In a way, Docker is a bit like a virtual machine. But unlike a virtual machine, which creates a whole virtual operating system, Docker allows applications to use the same Linux kernel as the system that they’re running on. It only requires applications to be shipped with things not already running on the host computer. This gives a significant performance boost and reduces the size of the application.

Version Control and Component Re-use

You can track successive versions of a container, inspect differences, or roll-back to previous versions. Container re-use components from the preceding layers, which makes them noticeably lightweight. Sounds like Github right ? Yep, Docker is an open source project as well.

Key elements we were looking for ?

• Isolated: No other project should affect it and it shouldn’t affect any other projects. For example, if one project uses Python 2.7, that shouldn’t affect another project which uses Python 3.4.
• Repeatable: Write it once and have it work each time. I don’t want to spend time tweaking it for each user.
• Production Replica: Be as close to production as possible.
• One Box environment: Easy to start/stop and manage.

All that sounds great. But how do we achieve this?

Here’s What We Did

We used Boot2Docker (A VM host to run docker daemon as it only works on Linux kernel) for Mac OS X and Fig, a orchestration tool for building, launching and linking Docker containers that moves all the configuration required to orchestra Docker into a simple clear fig.yml file. We also used a simple bash script over fig commands to increase usability.

We automated each and every step to setup the environment and we can now start our stack by running a single bash script. Now each developer has a 16 GB machine where boot2docker VM is using memory around 5 GB. The number of containers we can run on this is close to 25 including db containers with minimal setup.

Here’s a sample Dockerfile:

FROM python:2.7.9
COPY requirements.txt requirements.txt
RUN pip install --quite -r requirements.txt

A sample fig file:

redisdb:
    image: redis:2.8.19
apiserver:
    build: .
    command: python apiserver.py
    volumes:
      - .:/code
    ports:
      - ":8080"
    links:
       - redisdb

Now one can simply run fig up and your api server is running. In future if you want to scale your api server then you can do the following:

$ fig scale apiserver=2

And voila, you have two services running for api server

More Info on Docker

To get, the best place is Docker’s official document and their online interactive tutorial . But here’s a list of some commands I personally find very useful:

$ docker ps  
Lists all Docker processes running
$ docker help
To list all available options 
$ docker exec -it <container-id> bash 
To go inside already running container
$ docker rm <container-id>
To remove a container
$ docker rm $(docker ps -a -q)
To remove all containers
$ docker rmi <container-id>
To remove images of the given container
$ docker rmi $(docker ps -a -q)
To remove all images 

You can read about how to setup boot2docker on Mac OS X here

Other Notes:

• Tooling for Docker is also a blossoming set of options.
• Fig (now named docker compose) is now owned by Docker, and is an easy to use orchestration tool for building, launching and linking Docker containers.
• Docker is also in a race to build their own ecosphere of tools around the core Docker engine. Such as Swarm for clustering and Machine for container and VM management.
• Citadel, Panamax, Flocker and Shipyard lets you compose, deploy and/or manage Docker containers and clusters in many different ways. Dokku lets you easily deploy your Git based applications into Docker containers.

Every week I seem to be finding new and amazing tools for Docker and containers in general.

Conclusion

Docker is a truly amazing project and represents a huge leap forward for advanced systems administration. It’s very powerful and has many use cases beyond what I’ve discussed here. My focus for evaluating Docker has been primarily around server setups delivering web applications.

We found that it’s great to setup for a staging environment but taking it to production will need a lot more expertise.