Reduce Unit Test Runtime Using GitLab CI Parallel

RSpec runtime before-after

Along with the development continues to grow, the unit test runtime could have increased, and at some point, it could be very long to wait.

Our current service that handles offline to online (O2O) transactions Mitra Bukalapak, starts having increased time with unit tests pipeline and this causes the time required for deployment to increase as well. Then our team starts the initiative to find a way how to improve the deployment time window by reducing the unit test runtime.

We are looking for an approach that can implement as simple as possible without any major changes in our main code and yet still compatible if we decide to upgrade the framework in the future. Eventually, we are trying to implement the divide-and-conquer method to our unit tests using available resources.

There are three questions before we start paralleling the unit tests.

1. How to divide the unit tests?
2. How to run unit tests into parallel jobs?
3. How to collect results from each parallel job?

This article will show how our engineering team in Bukalapak reduce the RSpec test runtime using parallel Gitlab CI job process and report it using Coveralls.

Chapter 1: The Unit

Unit tests were meant to be autonomous, which each unit should be able to run independently without linkages between one another. Therefore, the first thing that we need to do is to divide unit tests into each separated group so later it can be executed by different jobs.

RSpec allows testing the test per file, such as

$ rspec spec/dandelion_spec.rb

It also allows multiple test files at the same time

$ rspec spec/camellia_spec.rb spec/dandelion_spec.rb spec/hydrangea_spec.rb

With this feature, we can write a script to list all of the test files and then split them into different ‘file groups’.

Fortunately, the open-source community already provide us a similar script to handle our task. We can use test-boosters to split the list of RSpec by using a simple command, you can read the full documentation here.

For example, if we want to divide test files into five groups and run the first group, we can execute a command as below

$ rspec_booster --job 1/5

Chapter 2: The Job

Gitlab CI provides us simple configuration to run parallel jobs on multiple GitLab Runners. You can see full documentation from Gitlab here.

If we want to create N parallel jobs, we just need to add parallel: N and the jobs will be named sequentially from job_name 1/N to job_name N/N . Each parallel job also will have CI_NODE_INDEX and CI_NODE_TOTAL predefined CI/CD variable set.

RSpec Gitlab CI job example

From the job above, we will have 15 parallel jobs running different rspec_booster command

15 parallel jobs example

You might notice there is another command with artifacts coverage directory

mv coverage/.resultset.json coverage/.resultset-$CI_NODE_INDEX.json

That command is required for the reporting data on the next stage.

Chapter 3: The Report

After the parallel jobs are done. The next step is to collect the result and combine it into one. RSpec resultset usually named as .resultset.json. That file stored a list of tested files along with a covered line of code. From the previous stage, we should have 15 resultsets from 15 parallel jobs

coverage
├── .resultset-1.json
├── .resultset-2.json
├── .resultset-3.json
├── .resultset-4.json
├── ...
├── .resultset-14.json
└── .resultset-15.json

We can write a script to combine and calculate all of those results. Fortunately, SimpleCov provides a method SimpleCov.collate to fetch them all and merge them into a single result set. You can read the full documentation here.

Here is our example script ./bin/calculate_coverage

Calculate coverage script example

Now for the job to report the final result, we only need to run the script we made before

Report coverage Gitlab CI example

Coveralls report example

Chapter 4: The Finale

With this method, we were able to reduce unit test runtime from ~14 minutes to ~3 minutes and that is about 80% time reduced.

RSpec using 1 single job

RSpec using 15 parallel jobs

We found some problems while paralleling the process, for example, some tests are required others to finished first or executed at the same job. Therefore, we need to group some tests at the same job and test-boosters already provide feature Split Configuration to do this.

However, grouping some test are not a good solution. That means our unit tests are not independent enough. The ideal solution is to check the failure test and refactor it to an autonomous test, but it will require more time and changes. For now, we can use this Split Configuration method to achieve reduced time as soon as possible and refactor the unit tests gradually since we already know which tests are dependent.

We also need to find the sweet spot of the total parallel job. In our case, even if we increase to 50 parallel jobs, the time required is almost the same as 15 parallel jobs. Using more jobs also means using more resources of Gitlab Runner which can affect other pipelines.

The tech stack here is demonstrated using Gitlab CI, RSpec, SimpleCov, and Coveralls. However, the same concept should be applicable to other libraries or programming languages. This article is just a proof of concept parallel test on multiple GitLab runners, and you can always try it on your own configuration as you prefer and discover new things through it.