CI: Build performance testing with github action

If DevOps and CI/CD is already something fully adopted in your company I am sure you have some automated test already, but do you test your performance?
If not you should consider adding performance testing in your release train.

I will present how I use GitHub action to test performance on one of my npm libraries. It gives tools to check if you have performance regression.

Write performance tests

As you can imagine, the first thing is to write performance tests.
For that and because I use typescript on this library I used benchmarkjs framework to write it.

A basic performance test will looks like this:

import Benchmark = require('benchmark');
const suite = new Benchmark.Suite;
suite
    .add('some test case', () => {
        // here call the code you want to test.
    })
    .on('cycle', event => {
        // Output benchmark result by converting benchmark result to string
        console.log(String(event.target));
    })
    .run();

If you are not using Typescript/Javascript, note that the GitHub action is compatible with:

• cargo bench for Rust projects
• go test -bench for Go projects
• benchmark.js for JavaScript/TypeScript projects
• pytest-benchmark for Python projects with pytest
• Google Benchmark Framework for C++ projects
• Catch2 for C++ projects

Why using a benchmark library

Because it is always a nightmare to make it in the code, these libraries are running the same code X time to be sure to have more relevant stats at the end.

Set up your GitHub Action

The GitHub Action is Continuous Benchmark and it is available in the GitHub Action marketplace.

What does this GitHub Action?

1. It runs your performance test
2. It stores the result in your gh-pages.
3. It compares it to the last result.
4. If there is the performance threshold is reach, it comments your commit.

1. Create your gh-pages branch

At first, you need to create a branch for GitHub Pages if you haven’t created it yet. This is used to display your dashboard.

# Create a local branch
$ git checkout --orphan gh-pages
# Push it to create a remote branch
$ git push origin gh-pages:gh-pages

Don’t forget to activate your GitHub Pages for this project on this branch. For that, you should go to your project settings.

2. Create a GitHub personal access token

You should create GitHub personal access token with the scope repo we use it to push on the gh-pages branch.

Do forget to copy it when it is displayed 😉.

3. Create your workflow

Create a file .github/workflow/benchmark.yml in your project.

This file should contain something like this:

name: Benchmark
on:
  push:
    branches:
      - master

jobs:
  benchmark:
    name: Check performance regeression.
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v1
      - name: Run benchmark
        run: npm install && npm run bench | tee output.txt
      - name: Store benchmark result
        uses: rhysd/github-action-benchmark@v1
        with:
          name: Benchmark.js Benchmark
          tool: 'benchmarkjs'
          output-file-path: output.txt
          github-token: ${{ secrets.PERSONAL_GITHUB_TOKEN }}
          auto-push: true
          alert-threshold: '130%'
          comment-on-alert: true
          fail-on-alert: true
          alert-comment-cc-users: '@thomaspoignant'

The important things in this file are:

• on part, as you can see we are checking performance only on the master branch. We are doing it to have an understandable graph at the end, if we do it on multiple branchs/pull requests, the result will be messy.
• run part, is the command you should call to execute your bench and to output your results in a file.
• alert-threshold, this is the threshold to comment your commit when the performance decrease.
• alert-comment-cc-users, each time there is a comment on a commit it will ping this user also.

4. How does it look

After the first workflow run, you will get the first result on https://you.github.io/repo/dev/bench like this.

And if your threshold is reached you will have a comment directly in your commit and it will look like this:

Conclusion

I use this system to detect performance issues on a library to test basic algorithmic regression. But you can use it to call an HTTP endpoint or doing a more complex logic that you want to test and monitor.

Add performance tests as much as you can. It will help you and your team, to have a better product and to be quick to detect if something is taking longer than usual after a release.