How Knapsack accelerated our test suite by 50%

By Mohit Choudhary.

At Scripbox, we continuously strive to keep our CI pipelines quick. If not for this speed, it would be impossible to ship code to production multiple times a day, with minimal errors.

Our core application stack is powered by Ruby On Rails and we have setup Gitlab CI for our test infrastructure. Much like any growing project, the more tests we add, the slower the CI pipeline gets.

Our naive implementation took care of slicing and assigning specs to individual runners.

We had 5 different jobs running concurrently and tests were divided in to equal numbers. Some jobs ran faster than the others:

rspec-job 0 5   3 mins 25 secs
rspec-job 1 5   3 mins 21 secs
rspec-job 2 5   3 mins 1 secs
rspec-job 3 5   13 mins 5 secs
rspec-job 4 5   8 mins 2 secs

The pipeline took 13 minutes to complete because one job took as much time. We needed a smarter way to distribute our tests so that they would take more or less the same to be completed, thereby bringing down the average time spent by the pipeline.

We started looking for ways to speed up our test pipelines and found the following options:

parallel_tests

parallel_tests works well when you run your suite in a single host. It runs the tests in parallel on multiple CPU cores. Since our Gitlab runners are small EC2 instances with just a single core, we opted for something that could work in a distributed fashion across the network.

knapsack

Knapsack splits tests across CI nodes and ensures they complete in more or less the same amount of time.

Installation

1. Add Knapsack to Gemfile.

We grouped it under :ci since it’s not required for development:

2. Bind the Knapsack adapter in spec_helper.rb :

CI is a predefined environment variable exposed in Gitlab CI, which ensures this happens only in CI, excluding other environments (eg. dev)

3. Finally add the following lines toRakefile :

How it works

Knapsack maintains a report which logs the time taken by each test. We can create the report by running:

KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

This creates knapsack_report.json which we need to cache in the gitlab.

Now we can run the test in CI as follows:

CI_NODE_TOTAL=2 CI_NODE_INDEX=0 bundle exec rake knapsack:rspec
# CI_NODE_TOTAL is total number of rspec jobs you intend to have.
# CI_NODE_INDEX is the current job number

Knapsack will refer to knapsack_report.json to split the tests.

We have an overhead of maintaining knapsack_report.json. When significant changes occur in our test suite, we need to manually generate the report again. Inspired by how gitlab addresses the same issue, we have now automated the report generation inside Gitlab CI.

Broadly these are the steps:

1. We cache the Knapsack report, so that it is available between runs.
2. Prior to the test stage we have a retrieve knapsack stage. We retrieve the report from cache and expose it to other jobs in the same pipeline as artefacts. Here’s the excerpt from our gitlab-ci.yml :

This is how our rspec job looks:

For each node we generate the Knapsack report separately. Since it has cached the main Knapsack report (exposed from previous stage as an artefact), Knapsack is able to divide the test equally among the nodes.

3. Finally we merge all those separate reports into one and cache it for subsequent runs.

Here is script/merge-reports :

And the results look promising already:

rspec-job 0 5   8 mins 10 secs
rspec-job 1 5   8 mins 21 secs
rspec-job 2 5   7 mins 30 secs
rspec-job 3 5   6 mins 5 secs
rspec-job 4 5   8 mins 40 secs

Conclusion

Before using Knapsack, it took an average of 16 minutes to complete over 3500 tests for a pipeline consisting of 8 runners working in parallel. This has now dropped to 8 minutes.

With the use of Knapsack, we already have the power to scale our testing infrastructure horizontally. We now intend to make the test pyramid smarter; say have a faster unit stage before the functional stage as this will improve the turn-around time.

We are unlocking engineering challenges, the kind that come frequently, and with scale. If you think these are the kind of challenges that you would love to solve, do check out our open positions.