Improving/Using our CI Pipeline

Background/Intro

Published in

SENSEI Developer Blog

5 min readJul 28, 2016

For as long as I have been in my current position the importance of testing has been stressed. My first few weeks at Cascade Energy included mainly writing and reading PHPUnit tests for our client data import pipeline. One-hundred percent unit test coverage was not an ideal to shoot for, but essentially a requirement. We also had a continuous integration (CI) pipeline in place in the form of a Jenkins server (since replaced by Travis). We thought we were doing great.

Random developer: Do you have test coverage?
Me: Of course, 100% unit test coverage.

The fact that testing was not only encouraged, but expected, made writing tests less stressful. As unit testing became more familiar, the tests themselves started to seem more obvious and less meaningful. It wasn’t that unit tests were pointless in their own right, but they didn’t feel like they solved the problem that our group wanted to solve.

Our ideal view of software testing meant security and lack of fear to make bold decisions and be creative, but our tests were doing the opposite. Our wish was for a testing harness that would test overall system functionality. This would allow us to modify the internals of a code-base and have a way to make sure that we didn’t screw anything up.

This ideal view was often discussed, but there were always barriers to entry like databases and networking that forced us to put everything on hold.

Docker to the Rescue!!!

Roughly a year ago, our team started moving toward a distributed service-based application architecture. This meant heavy use of Docker, which turned out to be the key in moving one step closer to our hope for simple, fast, developer-friendly, and safe tests of system components.

Story/Use-case

Our team recently began work on a new system component: user access and permissions. This system is designed to allow different levels of access to different system components for our users. The complexity of the components in our application necessitated a complex permissions system.

The permissions system, which will be discussed in greater detail in a following post, is essentially a graph of system objects with the graph edges representing access and level-of-access between the objects. The complexity of the design necessitated that every member of our team gain a deep understanding of the system before implementation. This was key as the first implementation appeared to be functioning properly until we tested specific use cases.

As we refined the implementation and continued to try to break it, we realized that, due to both the complexity and importance of the permissions system, we needed a way to know that changes we made were not breaking desired functionality.

I took it upon myself to begin work on an initial version of an end-to-end test of the system. This meant solving several specific problems:

Creating a Temporary/Development Database
Connecting the System and the Development Database
Creating a functional test
Adding E2E tests to CI Pipeline

Development Database

Under the hood of the graph used for our permissions system is Elasticsearch. In our production environment, there are hundreds of thousands of graph edges and nodes contained in a fairly beefy multi-node cluster. This is great for production, but too costly and time-intensive to set up for a simple development/test environment.

I decided that for a first pass, a single-node Elasticsearch cluster would be sufficient for the type of testing that I wanted. I used the standard Docker image of Elasticsearch and installed my favorite plugin, kopf. I did this using the following Dockerfile:

FROM elasticsearch:latest

RUN bin/plugin install lmenezes/elasticsearch-kopf

Networking

After getting a sample database running, I needed to wire it up to the permissions system. For this I decided to use docker-compose and it’s container linking feature. All that I needed was a docker-compose.yml file to specify my required set up, and a corresponding directory structure:

Directory structure:

project-root/
  development/
    elasticsearch/
      Dockerfile
    docker-compose.yml

docker-compose.yml

version: '2'
services:
  permission:
    build: ..
    ports:
    - "9000:9000"
    volumes:
      - ./default.json:<application-config-directory>/default.json
    links:
    - elasticsearch
  elasticsearch:
    build: elasticsearch
    ports:
    - "9200:9200"
    - "9300:9300"

To use docker-compose, I needed just one simple command:

$ docker-compose up — build -d

This creates an Elasticsearch node and a copy of the permissions system, both running in containers on my local machine. Because of the magic of Docker linking, the permissions container can communicate with the Elasticsearch container via http using the Elasticsearch container-name “elasticsearch” as a DNS entry. For example, to check the connection to Elasticsearch from inside the permissions container:

$ curl http://elasticsearch:9200
{
  "name" : "random-name",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.3.4",
    "build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
    "build_timestamp" : "2016-06-30T11:24:31Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

Creating a functional test

It was all well and good that I had the permissions system talking to an Elasticsearch cluster, but if the database had nothing in it, there would be nothing to test. I took one of our use cases that had broken a previous iteration of the system and wrote a script that populated the database with the corresponding graph. I then wrote a quick mocha script to query the permissions service and compare the results to the expected output. I added a new script to the project’s package.json so that I could easily call the functional test separately from the unit tests.

Adding End-to-End Tests to Continuous Integration Pipeline

Our team uses TravisCI for continuous integration. This makes integration extremely straightforward as the only thing to change about the permissions system was the .travis.yml file. The changes required included:

Don’t run Travis build in a container

sudo: required

Require Docker as a service and upgrade to latest version

services:
  - dockerbefore_install:
  # Update docker-engine using Ubuntu 'trusty' apt repo
  - >
    curl -sSL "https://get.docker.com/gpg" |
     sudo -E apt-key add -
  - >
    echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" |
     sudo tee -a /etc/apt/sources.list
  - sudo apt-get update
  - >
    sudo apt-get -o Dpkg::Options::="--force-confdef" \
     -o Dpkg::Options::="--force-confold" --assume-yes install docker-engine
  # Sanity check
  - docker version

Upgrade to latest version of Docker-Compose (1.8.0-rc2 as of 7/27/16)

env:
  DOCKER_COMPOSE_VERSION: 1.8.0-rc2before_install:
  # Install required version of compose
  - sudo rm /usr/local/bin/docker-compose
  - curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
  - chmod +x docker-compose
  - sudo mv docker-compose /usr/local/bin
  # Sanity check
  - docker-compose version

Run Docker-Compose to create required resources

before_install:
  # Build required infrastructure
  - docker-compose --file development/docker-compose.yml up --build -d
  # Give Elasticsearch some time to initialize
  - sleep 10

Run both unit tests and new end to end tests

script:
  # initialize environment (create elasticsearch index, insert graph)
  - ...
  # Run unit tests
  - npm test
  # Run End-to-End tests  
  - npm run functional-test

Clean up

after_script:
  # Clean up after ourselves
  - docker-compose --file ../development/docker-compose.yml down

Conclusion

Overall the process took a few days of fiddling to get everything just right. The process is not complete, but we have a good start toward a full test suite that should solve the problem that we have always wanted our tests to solve: providing a safety net so that change is encouraged rather than feared.

—
Nicholas Warlen
Software Engineer @ Cascade Energy Inc.
nick.warlen@cascadeenergy.com
https://github.com/CascadeEnergy
https://github.com/nwarlen