Paving the Way, part 3 — Automating tests and builds

Since we are aiming to bring down the deployment times and up the reliability of it, we need to get rid of the fleshy bit between the chair and the keyboard (sometimes affectionately referred as “human”) and automate everything we possibly can. One of the key pieces in this puzzle is Continuous Integration, which… well, continuously runs tasks against the code base, such as testing or building release packages.

Read the previous part here: Paving the Way, part 2 — Docker all the things!

Photo by Louis Reed on Unsplash

Continuous Integration

For Continuous Integration & automation we originally started with Bitbucket Pipelines but ended up scrapping that due to various problems with build minutes. Nowadays we are running in CircleCI and while it is not perfect, we have had no major issues so far, it even integrates with Bitbucket to update the build status to the commit details. So far so good. Few words of advice though:

Use contexts. In CircleCI, whenever you have a setting that is common for multiple projects AND which aren’t classified similar to nuclear launch codes (aka. never ever should be available to anyone) you should use contexts, which essentially are just collections of environment variables that are passed to the container running specific task in CircleCI. The UI nicely hides the values themselves however, be aware that they are obviously available on build time so the values can be echoed out if one so desires.

We are using CI for running tests & building Docker images, Lambda release packages and infrastructure-as-code source packages.

Visualization of CI tasks

Testing

One of the traditional use cases (and arguably the most important) for Continuous Integration is testing automation, which obviously is something we do as well. Every time something is committed to any source repository branch, a bunch of tests (such as system and style tests) are run against said code to catch any possible problems as soon as possible. We tend to favor system testing over unit testing in order to keep test suites easier to manage & maintain and only build unit tests for the most crucial parts.

On top of the traditional testing processes, we also have full-stack integration tests (where available) which are run before and after every deployment. These tests are however run separately from the CI as they need to be able to access the isolated environments where staging and production setups are located.

Automated Docker builds

Docker builds are run automatically whenever semver (ie. v1.0.1) git tag is pushed to the repository. Semver versioning scheme is used as it makes it easy to understand the scope of the builds from major builds to new features to small fixes (v{major/breaking change}.{new feature}.{fix}). Git tags also make it easy to see what the build contains by checking the contents of the said git tag in the code repository.

For Node.js projects the process goes as follows:

1 . Update version number in package.json and create matching git tag with npm version by running npm version

npm version major/minor/patch

2. Push updated package.json and generated git tags

git push origin master && git push origin master --tags

Alternatively set git to send tags automatically with every push
(git config --global push.followTags true), meaning you can omit the latter git push command.

Once the semver git tag lands to the repository, CircleCI will automatically build the Docker image, tag it with the version number and push it to Docker Cloud, where it is then available for consumption such as deployment to AWS EC2 Container Service (ECS), where we run our Dockerized services.

As for AWS Lambda deployments, the process is the same except that Docker build and push steps are replaced by building the Lambda source zip file and uploading it to the deployment S3 bucket.

Automated infrastructure-as-code builds

One of the parts of the CI is also to automate the deployments of any infrastructure changes, such as new application releases. For us, this currently means building the infrastructure source package that the automation uses for deployments. This will be covered more in detail in the next part of the series.

Example

Example CircleCI config (.circleci/config.yml) with testing for all branches and Docker builds based on semver git tags.

default: &defaults
  docker:
    - image: node:10
version: 2
jobs:
  test:
    <<: *defaults
    steps:
      - checkout
      - run: "npm install"
      - run: "npm test"
  build:
    <<: *defaults
    steps:
      - checkout
      - run: "npm install"
      - run: "npm test"
      # Setup local & remote Docker clients for build/push
      - setup_remote_docker 
      - run: | # Setup local Docker client
          set -x
          VER="18.06.1-ce"
          curl -L -o /tmp/docker-$VER.tgz https://download.docker.com/linux/static/stable/x86_64/docker-$VER.tgz
          tar -xz -C /tmp -f /tmp/docker-$VER.tgz
          mv /tmp/docker/* /usr/bin
      - run: "docker login --username $DOCKER_USERNAME --password $DOCKER_PASSWORD"
      - run: "docker build --tag allermedia/$CIRCLE_PROJECT_REPONAME:$CIRCLE_TAG ."
      - run: "docker push allermedia/$CIRCLE_PROJECT_REPONAME:$CIRCLE_TAG"
workflows:
  version: 2
  run_test:
    jobs:
      - test:
          filters:
            tags: # Don't run on tags (builds have their own test)
              ignore: "/^v[0-9.]+$/"
  run_build:
    jobs:
      - build:
          context: "build"
          filters:
            branches: # Don't run based on branches
              ignore: "/.*/"
            tags: # Run build job only on semver tags
              only: "/^v[0-9.]+$/"

Note that testing step ignores semver tags since build has it’s own testing step, this is just to avoid spamming extraneous test steps to CircleCI tasks list on every build.

Future

The whole setup has been build with continuous deployment in mind. Eventually, this can easily be achieved by switching to building Docker images per merge to master and deploying it automatically. However, since we have yet to land on the exact integration testing setup we want to run (we are currently testing Puppeteer integrated to AWS CodePipeline), thus until that process is finished we are still running semi-manual deployment of the images by changing the version number in infrastructure-as-code files manually and pushing the changes to code repository where automation takes over.

Next part we focus on the infrastructure automation through infrastructure-as-code and how we deploy services.

— By Mikko Tikkanen, Technology Lead at Aller Media Finland