Moving our Wercker CI to the new Docker stack

Open Listings is engineering a better way to buy a home: https://www.openlistings.com

At Open Listings, we’ve been using Wercker, an awesome CI service (that’s free while in beta!), since day one. We were really excited when they announced they were moving their stack to Docker.

We’re sold on the promise of Docker, but hadn’t gotten our hands dirty with it yet. While Wercker has a good deal of documentation on their Docker-based stack, we couldn’t find much in the way of a guide for transitioning our old configuration to the new. We basically wanted an Ubuntu box with Ruby, Node.js, and Firefox and PhantomJS for Capybara tests.

Here is the blog post I wish I’d been able to find for those who want to transition their Wercker config to the new Docker stack.

How it wercks

1. Create a Dockerfile that encapsulates the instructions for setting up your container, similar to your install script if you ever created a box on the old wercker stack.
2. Use the docker binary to build an image from the Dockerfile.
3. Tag the image and push the image to docker hub — the image name needs to match the username and repo you’ve created in docker hub for everything to magically work.
4. Modify your wercker.yml to specify the box as your dockerhub repo and your services as docker hub repos.
5. Configure your application with the new Wercker environment variables so that it can connect to those services.

Prerequisites

Setup docker and boot2docker

Get docker and boot2docker working. On OS X, Homebrew works great for installing docker and boot2docker, which you’ll need to run a VM for running docker containers.

Add the environment variables that boot2docker prints out after you run:

boot2docker up

This makes it so docker knows how to connect to the VM.

If you’re on OS X you can also use Kitematic which gives you a GUI for some of these tasks and sets those environment variables for you in a special shell.

Setup a dockerhub.com account

Go to dockerhub.com, setup an account and make a new repository.

Login to your docker hub account on the command line with docker so that it has permission to push to your new repository.

docker login

Setup Wercker CLI tools

Install Wercker CLI tools — essential and awesome. Local builds make it super easy to work the bugs out of your image since you can just run what wercker runs locally without having to push up and watch your builds compile through the web interface each time.

http://wercker.com/downloads/

To build locally:

wercker build

Modify wercker.yml and add services

Setting up the two services we use — redis and mongo — was pretty painless. We changed our services section in wercker.yml to read:

services:
    - mongo:2.6.7
    - redis

You can find the names of the services you want to use by searching on dockerhub.com.

The concept is the same as before: Wercker is going to spin up boxes running these services and expose the endpoints to you. You just have to find the right environment variables so your app knows how to connect. To that end, follow the advice here: http://devcenter.wercker.com/docs/services/available-env-vars.html

“If you want to view which variables are exposed in your container while it’s running, run the env command in a script step in your wercker.yml file.”

We ended using these:

REDIS_PORT_6379_TCP_ADDR (redis host)
REDIS_PORT_6379_TCP_PORT (redis port)
MONGO_PORT_27017_TCP_ADDR (mongo host)
MONGO_PORT_27017_TCP_PORT (mongo port)

N.B. Wercker still sets the WERCKER environment variable to true so you can check in your application when you are running in the context of Wercker.

WERCKER_CACHE_DIR is also still the directory you can use to save stuff like installed gems between deploys. Wercker added a new step at the end of the build process called “Store” that manages this for you. The “bundle-install” task in wercker.yml still magically bundle installs for you in the optimal way.

Docker crash course

Create your Dockerfile in a new directory in your project, not your project root, otherwise the entire project will get sucked into the image and your image size will be big. Keep your Dockerfile under source control.

The Dockerfile starts with a pre-existing image that you can modify via running commands. We wanted to use Ubuntu 14.04.2 as our base image, so the first line is:

FROM ubuntu:14.04.2

apt-get install anything else, you’ll have to tell apt-get not to wait for Y/N responses from the user

RUN apt-get install gcc -y --force-yes

Then add any other software you want. Between RUN commands, the current working directory isn’t preserved, so either work with that or string together a bunch of commands with &&.

Here’s our Ruby 2.2.0 install:

RUN cd /root/src && wget http://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.0.tar.gz && tar xzvf ruby-*.tar.gz && cd ruby-2.2.0 && ./configure --with-readline-dir=/usr/include/readline --with-openssl-dir=/usr/include/openssl && make && make test && make install

Then tell docker to build this image

docker build .

You should see something like this… Keep track of the SHA

Successfully built 05879b638edc

Then create a tag, where username/repo are your details from docker hub

docker tag 05879b638edc username/repo

The tag name defaults to “latest” if you don’t specify. Also, you may need to use the -f option if you’re modifying your image and have already tagged something “latest”. You should see no output.

Finally, push the image up to docker

docker push username/repo

(Probably will take a while)

Some other useful commands are:

docker images

(See what images you’ve created already.)

docker run -i -t username/repo /bin/bash

(Create a container from the image and open a bash shell inside the running container)

Rough edges

Compiling PhantomJS & OOM

When installing PhantomJS, we were getting OOM errors “virtual memory exhausted: Cannot allocate memory” until we added the “jobs” parameter that tells the install script to only use the specified number of cores instead of all the VM’s cores.

RUN cd /root/src/phantomjs && git checkout 2.0.0 && ./build.sh --confirm --jobs 2

Environment variables in wercker.yml

You used to be able to specify environment variables in wercker.yml in a block called “env”, but it appears that that went away as wercker complains:

Error parsing your wercker.yml: Invalid extra key in config, p env is not a pipeline

For firefox to run correctly for our capybara tests on ubuntu, we need an environment variable DISPLAY=”:99.0”

The easiest thing is to add this as a step in your Dockerfile or via the wercker web ui for your project, otherwise you’ll see something like:

Selenium::WebDriver::Error::WebDriverError: unable to obtain stable firefox connection in 60 seconds (127.0.0.1:7055)

This was a hard one to debug because it just looked like the tests were taking a long time to run. We ended up commenting all our capybara tests but a very simple one (visit ‘/robots.txt’) and that illuminated the real source of the problem.

bower-rails rake task

We use the bower-rails gem to manage our bower packages. The rake task complained about running as root. First, we thought about setting USER in our Dockerfile to another user, but it turns out that Wercker requires the user to be root, so this didn’t pan out. You may have to adjust other steps in your script that refuse to run as root — our bower-rails step became:

RAILS_ENV=test bundle exec rake bower:install['--allow-root']

Docker is not an Amazon AMI

When you exit from a docker container on the command line, you lose all the changes you made, so it’s not a good idea to install software on the command line and then save your image like you would do with Amazon AMI’s on EC2. You’ll want to make all your changes in the Dockerfile and just use the command line for testing things out and debugging.

Wercker specific requirements

If you’re seeing error messages when you run “wercker build” like:

Step failed: setup environment

then you probably have to tweak your Dockerfile to set the ENTRYPOINT and USER that wercker wants (root).

ENTRYPOINT ["/bin/bash", "-c"]

This github issue is relevant if you have this problem: https://github.com/wercker/support/issues/24

If you don’t specify the ENTRYPOINT, it uses the one from the FROM image, which for ubuntu is the one you want anyway.

Finally

We ran some Wercker builds locally against our docker image, adding software to it as we needed, until we worked through the bugs above. Docker caches the intermediate builds as you add to your Dockerfile and you can easily jump into a console to see what’s going on if your build script requires fine tuning. This makes it a lot faster and easier to update your image as you want to add or upgrade your box’s software.

If you have a lot of tests, I’d recommend commenting out most of your tests at first and gradually bringing them back in as you’ll be able to iterate more quickly that way. Get your unit tests running, then modify your Rakefile to only run your integration tests until those are all running.

Here’s our final Dockerfile:

# OpenListings Wercker Image
FROM ubuntu:14.04.2
MAINTAINER OL Engineering <engineering@openlistings.co>
LABEL Description="OpenListings CI"
# slim down the image - http://www.hnwatcher.com/r/650964/Slimming-down-Docker-containers-Intercity-Blog
RUN apt-get clean
RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN apt-get update
RUN apt-get install gcc g++ make curl zlib1g zlib1g-dev vim openssl libcurl4-openssl-dev libreadline6-dev unzip libffi-dev -y --force-yes
RUN apt-get install libssl-dev libpcrecpp0 libpcre3-dev wget git libreadline-dev libqtwebkit-dev xvfb imagemagick -y --force-yes
RUN mkdir /root/src
# phantomjs 2.0
RUN apt-get install build-essential g++ flex bison gperf ruby perl libsqlite3-dev libfontconfig1-dev libicu-dev libfreetype6 libssl-dev libpng-dev libjpeg-dev python libx11-dev libxext-dev -y --force-yes
RUN cd /root/src && git clone git://github.com/ariya/phantomjs.git
RUN cd /root/src/phantomjs && git checkout 2.0.0 && ./build.sh --confirm --jobs 2
RUN mv /root/src/phantomjs/bin/phantomjs /usr/local/bin/phantomjs
RUN rm -rf /root/src/*
# firefox
RUN apt-get update && apt-get install firefox -y --force-yes --fix-missing
RUN echo "DISPLAY=:99.0" | tee -a /etc/environment
# node
RUN cd /root/src && wget http://nodejs.org/dist/v0.12.7/node-v0.12.7-linux-x64.tar.gz && tar -xzvf node-v0.12.7-linux-x64.tar.gz && sudo mv ./node-v0.12.7-linux-x64/lib/* /usr/lib/ && sudo mv ./node-v0.12.7-linux-x64/bin/* /usr/bin/
RUN rm -rf /root/src/*
RUN npm install -g bower
# ruby
RUN cd /root/src && wget http://cache.ruby-lang.org/pub/ruby/2.2/ruby-2.2.0.tar.gz && tar xzvf ruby-*.tar.gz && cd ruby-2.2.0 && ./configure --with-readline-dir=/usr/include/readline --with-openssl-dir=/usr/include/openssl && make && make test && make install
RUN rm -rf /root/src/*
RUN echo "gem: --no-rdoc --no-ri" >> /root/.gemrc
RUN gem install bundler -v "1.10.6"
RUN echo "RAILS_ENV=test" | tee -a /etc/environment
RUN echo "RACK_ENV=test" | tee -a /etc/environment

The openlistings/web image is public on docker hub, so feel free to use it if you like:

openlistings/web Repository | Docker Hub Registry - Repositories of Docker Images

For reference, here’s a link to the old Wercker box that we had been using, which was a modified version of their “web-essentials” box: https://github.com/openlistings/box-ubuntu12.04-ruby

And check out Open Listings, the easiest and least expensive way to buy a home in California!