DevOps at the Province of Nova Scotia

By Steven Zinck, Paul Badcock

Introduction

DevOps is the practice of breaking down silos between Development and Operations teams. DevOps promotes a culture and practices where Dev and Ops teams have open communication and collaboration. This article explains how the Linux Ops team and one of the Development teams at the Government of Nova Scotia were able to implement DevOps practices and realize its benefits.

The Beginning

The Linux Ops team was asked to host a Ruby application built circa 2006. We’re a Red Hat Enterprise Linux shop, provisioning the newest release of RHEL 7 and the Ruby app required gems that are only compatible with RHEL 6 and older. So, we had two options — provision a new RHEL 6 VM — something we haven’t done in over a year, or take this opportunity to containerize the application and use it as a proof of concept. Although we’ve been using containers for over two years in our Puppet CI environment, and have containerized some of our own management apps, this was our first client application to containerize.

Ops Digs into the Application

Since the Ruby code for the application was already in our GitLab, we had easy access to it so we could begin to understand its functionality. How does authentication work? How does SSL work? Where are assets stored? Exactly which gems are required? Does the system send email, and if so, how?

As we started to peak into the application, we found the answers to all of these things and a lot more. We were able to make a couple improvements to the application, for example, we were able to remove hard-coded values and switch to variables. In particular, we were able to expose database connection strings which can be leveraged by Docker Swarm.

The Docker Image

Now that we had a better understanding of how the application works, we started working on the container to host the application. We started with a base image of RHEL 6.9 and began layering on the dependencies and the application itself. Since the Development team is naturally very knowledge about their application, we collaborated closely with them on this process.

Automatically Building and Deploying

Once we had an image we were happy with, it was time to configure Docker Swarm and configure GitLab CI to push the image to our Docker registry.

I’ve included the relevant piece of our CI configuration below. As you can see, we’re tagging the docker image with the last commit # and pushing it to our internal registry.

build_image:
image: docker:1.12
stage: build
script:
- docker build -t ${DOCKER_REGISTRY}/${NAMESPACE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHA}
- docker push ${DOCKER_REGISTRY}/${NAMESPACE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHA}

Now that the image is up on our registry, we can tell Docker Swarm that a new image is available. Swarm will automatically pull down the new image and reload the application with less than 5 seconds of downtime.

DOCKER_HOST="${DOCKER_DEV_HOST}" docker service update --image ${DOCKER_REGISTRY}/${NAMESPACE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHA} ${CI_PROJECT_NAME}_app_1

Automating Security Scanning (DevSecOps!)

In addition to building the image, we also run a battery of security tests against the application code, the operating system, and application in its running state.

As you can see from the pipeline, after the image is built, we run a static code analysis using Brakeman. Brakeman tests the code for security issues, and since it’s a code analysis tool, the application doesn’t need to be running. After the code scan, we run Red Hat’s atomic scanner against the image. This tool will notify us of any known security issues in the operating system. Finally, we can deploy the application and then run Arachni to test the application in its running state.

Benefits of DevOps

We’ve several benefits from this approach:

  • The Ops and Dev teams worked closely together, each learning about the others domain expertise. As Ops discovered issues with the application, we were able to make code changes that were peer reviewed by the Dev team using the Git Flow development model.
  • The time to delivery for the application has improved drastically, and a framework has been established that existing, new and 3rd party staff can all leverage
  • Lower failure rate — if a new vulnerability is introduced into the stack, we’ll know
  • Fixes can be applied on-demand by Dev without Ops involvement
  • Recovery of the application is now as simple as two clicks
  • Dev and Ops both understand how the application functions and have a blueprint of its architecture in the Docker configuration

Next Steps

We’re actively collaborating with other Development teams across government to implement DevOps style practices. From a technology perspective, we’re aggressively working towards improving our technology stack so that we can improve business value for our customers.