How we use GitLab at the Province of Nova Scotia
In 2015 the Unix operations team at the Province of Nova Scotia decided to implement GitLab for source control and Continuous Integration & Continuous Deployment. This was the beginning of our foray into DevOps practices. This article describes our automated testing, integration and release of Puppet code.
A source control management (SCM) system allows the user to “commit” code, documentation and other system artifacts such as configuration files to a central location. Each change results in a new version of the file, and previous versions of the file remain available on the SCM. Restoring a previous version is quick and easy.
We needed a way for multiple sysadmins to be able to work on code without colliding with one another. We also needed a way to vet changes through a peer review process. GitLab makes this easy thanks to it’s support of branching and merge requests. Branching allows a sysadmin to create an individual copy of the production code (“master”) and work with it in isolation — this allows multiple team members to be working on the same production code base without being concerned about conflicts between their work.
As we built out more of our infrastructure with Puppet, we needed an automated way of testing our code. Over time our test strategy has evolved to include automated syntax checking, linting, unit and integration tests. Manual testing was not sufficient, as it was often forgotten about and was very time consuming. Automated testing solved that — for every code commit, the test pipeline is executed. A complete test cycle currently takes under 5 minutes.
On each code commit to a branch other than master, the following test pipeline is kicked off by GitLab CI:
If at any point a job fails, the pipeline stops and the sysadmin is notified. One of the great features of GitLab CI is its tight integration with Docker — each of the jobs above is run inside its own isolated container. The syntax-lint-spec job verifies that the Puppet syntax is good; linting confirms the code conforms to best practices; and spec confirms that logically the code functions as designed.
The test-kitchen jobs are a full suite of ServerSpec tests. We automatically provision 4 containers that represent our 4 most common configurations. Our Puppet code is applied to each container to verify that it will work in our production environment. This acts as a full regression test each time a code commit is made, and ensures that there were no unintended problems introduced. It gives us confidence that the code is actually doing what it’s intended to do.
Once all of the tests pass, the sysadmin can submit a merge request for their branch, and it will be reviewed by a senior staff member before reaching production. This is an important part of our workflow, because it gives junior staff the confidence that a more senior member of the team will review and approve a change before it reaches any of our servers. If the merge request is accepted, the branch will be merged into master and at that point GitLab-CI will push the code to our Red Hat Satellite and Puppet Enterprise servers where it will be deployed to our environment.
You can find the configuration files (Dockerfiles, .kitchen.yml, .gitlab-ci.yml and Satellite push script) at our GitHub.
The implementation of our system automation strategy and the toolset we selected has proven itself many times. We are spending less time fighting fires due to the streamlined & tested nature of our deployments and have earned the confidence of our clients.
The Road Ahead
In upcoming articles, we’ll write about the CI/CD process we built with Communications Nova Scotia that allows their development team to deploy and rollback their Dockerized application environment on-demand. We also plan to write about our automated test strategy for Red Hat Ansible.