Building an efficient Gitlab CI

Here at Octopus Wealth we are building react applications, such as the one powering octopuswealth.com. Each change to these apps is built and tested in CI before being merged into the master branch. The master branch is then deployed into production.

It is vital that the CI process is quick, as waiting for it to finish is a real productivity killer. For us this means that the entire CI process should complete within 5 mins.

Gitlab provides a inbuilt CI system that is configured via a declarative YAML file, called .gitlab-ci.yml, within the repository to be built. This makes Gitlab a very useful development platform.

Our fast efficient CI

Linting and testing

The purpose of the CI process is to give confidence that the proposed change works and is in a maintainable state. We consider the app to work if the tests pass and the production assets can be built. It is considered maintainable if the formatting is consistent, there are no linting warnings, and the code passes review.

To automate the tests, building, formating and linting we started with the following .gitlab-ci.yml file,

image: node:10-alpine

before_script:
- yarn install

script:
- yarn run format
- yarn run lint
- yarn run test
- yarn run build

this starts by installing all the required dependencies, then runs scripts to check the formatting, linting, testing and building.

Stage   | Execution time
------------------------
Install | 50s
Format  | <10s
Lint    | 10s
Test    | 10s
Build   | 70s
========================
Total   | 2m 30s

This is well within the 5 minute target, yet it can be made quicker.

Caching node_modules

Sadly the install part of the CI process takes almost a third of the time, yet it adds little value (as it says nothing about the quality of the change). It is also something that rarely changes, as whilst we upgrade dependencies regularly we do so less regularly than we make other changes. Therefore it makes sense to cache this part, using gitlab ci caching.

We can avoid most of the installation cost by caching the output, i.e. the node_modules folder. This results in gitlab storing the folder and our jobs downloading it before each CI job and then uploading a new cache after (to cache changes). This is simply done by adding the following to the .gitlab-ci.yml file,

cache:
  key: node-cache
  paths:
  - node_modules/

note the cache key is constant, which means the cache will be used for all jobs. This is fine as yarn install runs after and will make any branch specific installations.

With caching in place the install part now typically takes 1–2 seconds, thereby reducing the total time to around 1m 40s. It may be possible to be quicker still by executing the parts in parallel, which if the total time creeps up will be our next course of action.

We’ve written how we do continuous deployment in Terraforming GitLab & Heroku to deploy Dockerised apps.