Photo by Charl Folscher on Unsplash — I wish I was doing this kind of testing…

Fixing SimpleCov Inaccurate Test Coverage Outputs

An unfortunate journey (in Rails 6 and Ruby 3)

I love Ruby on Rails. So often, there is a gem that exists that you install and everything just works like magic. Unfortunately, this was not my experience with SimpleCov, a test coverage gem I was setting up for https://write-draft.com. When I followed the simplecov instructions for how to set things up, my output report showed zero coverage, even though I had a number of tests written. And that number was more than 0.

Screenshot of inaccurate test coverage report

Below is my journey of figuring out how to make it work. For reference, here is my environment setup:

MacBook Pro (13-inch, M1, 2020)
ruby 3.0.2p107
rails 6.1.4.1
simplecov 0.21.2

In an effort to not be those unfortunate recipe website (just show me the fucking recipe!), here is the solution.

Solution

• add the simplecov gem to your Gemfile in the :test group. Run bundle install
• set config.eager_load = true in environments/test.rb
• add this to your spec_helper.rb or test_helper.rb at the top

require ‘simplecov’
SimpleCov.start ‘rails’

• run rails t

I didn’t want to generate this report every time, so I made a few improvements.

Improved Solution

• add the simplecov gem to your Gemfile in the :test group. Run bundle install
• set config.eager_load = true in environments/test.rb
• use this nifty rake task (add to lib/tasks). Run with rails test:coverage

Rake task for running tests with coverage

With this solution, if you run rails t your tests will run without coverage. If you want to generate your coverage output, run rails test:coverage. This way made more sense to me because I don’t want to generate a coverage report every time I run tests (i.e. in my CI pipeline). If I’m not going to be doing it every time, better to only require simplecov when it’s going to be used.

The Journey

I started off by following the setup instructions:

• add the simplecov gem to your Gemfile in the :test group. Run bundle install
• add code to test_helper.rb (I use minitest)
• run rails test

WTF? I know my tests aren’t great, but they definitely exist.

Follow the internet rabbit trail of different solutions

First, I tried running with spring disabled (DISABLE_SPRING=1 rails t). No dice. Still 0% coverage, but I did learn how to disable spring should I ever want to for some reason.

Someone else suggested adding the require ‘simplecov' and SimpleCov.start to the beginning of the bin/rails file to make sure it was started prior to the tests being run. This felt pretty shady, but I was desperate so I tried it anyway. Nothing changed.

A third person indicated that I need to set config.eager_load to true for my test environment. It worked! 🎉

Screenshot of accurate test coverage report

Now, I had to figure out which combination of these things was necessary. I started eliminating until things didn’t work anymore. Turns out, setting config.eager_load = true is the only thing I needed. I went back and cleaned out my bin/rails changes and I stopped running with DISABLE_SPRING=1.

After a journey of an hour or more, the final fix is actually pretty simple. That’s usually the case, right?

After going through the normal install instructions for simplecov , set config.eager_load = true. That’s it. Hopefully, this saved you a little time :)

But do I really want to generate coverage every time?

I decided that I did not. Maybe you feel different. That’s fine. I also wanted to play with creating my own rake task, so I moved everything into a rake task:

Rake task for running tests with coverage

I learned a few things while doing this:

It is possible to extend rake namespaces

I do this with the :test namespace here. I can add the task test:coverage without altering any of the other tasks in the default :test namespace. Pretty sweet.

If I need a gem for a rake task, I can require it there

Maybe this one is obvious to more seasoned rails developers, but I’m still learning the ins and outs of things. In this case, I only need simplecov when I’m running this task, so I can remove it from my test_helper.rb, include it here, and everything works great!

There you have it. My unfortunately long journey to get simplecov working has come to an end. It was frustrating, but I learned a few things along the way. Now, when I’m wondering if I should feel bad about my test coverage, I can have a number tell me instead of a vague intuition 😄.

Hopefully, sharing this will make the process a little less painful for the next people who travel this same path. :)