Introducing SpecMetrics
A data-driven solution to improve your test suite
tl;dr
To fix our test suite (now almost 1-hour long and using re-run to handle randomly failing examples), we needed a more data-driven solution. So we built SpecMetrics, a work-in-progress tool to get insights from your test suite and fix it.
The long story
About 3 years ago, we started writing RSpec tests on our JobTeaser codebase. At the time, we had no continuous integration and some old tests that covered, well, probably nothing. We then started investing more and more time in our testing:
- quickly set up a CI (using CircleCI, which made it really fast),
- guidelines evolving toward a “no code without test” practice, in particular for bugs where non-regression tests are mandatory.
3 years later…
After 3 years, we may be proud of the work accomplished on our test suite:
- every code push is followed by a test build (about 2000 a month!),
- we will soon reach to a total of 6000 examples,
- our coverage (SimpleCov) is about 80%.
But we still, we’re not proud yet. Why?
We’re a fast-growing team (14 backend developers today and hiring). We think testing is hard, and we must recognise that with our current rhythm we have not been able to grow our testing culture and guidelines well enough. And we’re now facing two real hard issues:
- our test suite is now almost 1-hour long,
- we have introduced rspec-rerun to handle some randomly failing tests…
This state is not great, but the worse thing is that with about 6000 tests, you can’t fix it that easily. Almost every developer entering the team tried (voluntarily ;)) to optimise the test suite but well, there is no quickwin here. Sure we have options (like parallelising), but with a 200% increase in run time per year, we can’t continue on this path.
A data-driven approach to fixing tests
We are transforming ourselves into a data-driven company. This means we want to use data wherever it may be used to help us make good decisions, so it’s an evidence for business and product choices, but we’re using this state of mind for development too.
To fix our tests, we think we need to:
- have an idea of the repartition of run time per group of tests (acceptance, controllers, units… and even at lower levels),
- identify tests that were failing randomly and the ones that caused them to fail,
- monitor the increase in run time involved by added tests.
Introducing SpecMetrics
After reading some articles and looking at the great solutions from ForakerLabs (see this article and the associated rspec_profiling gem), we decided to ship our own solution since it was easy to reuse their solution and adapt it to our needs.
What does SpecMetrics do?
It plugs into RSpec to generate a JSON report of your test suite and send it to S3. The report contains:
- the list of all examples with their status, run time, number of requests and queries performed and the time they took,
- detailed information about the build (total time, git SHA, environment where it was run…).
What can I do with it?
With the information on all these builds (this includes builds run on the CI but also the ones run on development machines), we’re now working on extracting useful information. This is the goal of SpecMetrics Dashboard which uses Python data science tools. The current version is a work in progress, but you can already:
- access information for all build reports available on your S3 bucket,
- navigate by git branch,
- display run time statistics information for a specific run (including a drill-down on the 2 first levels of directories to get insight on where the most time is spent in the build and a detailed table for all levels).
Wanna try?
Two links:
- a gist to add SpecMetrics to your Rails / RSpec project and start generating reports to S3,
- SpecMetrics Dashboard to use the reports and get insights.
As said before, it’s a work in progress, we’re iterating to fill our needs, but if you want to use it, do not hesitate to leave a comment, here or on Github, so we may know how you are using it and if we can help you.
And if you want to work with us on our test suite, contact us at or have a look at our jobs, we’re hiring!
