Parallel test runs and coverage

At Marketfy we’re using CircleCI to run tests.

They have this great feature that runs your test suite in parallel without the need to do anything besides turning the feature on.

The only problem is that when your tests are run in parallel, you can’t “easily” measure the test coverage of your codebase.

What’s the solution?

1. Do some very complicated stuff, like manually splitting your test runs in several containers, ensuring they are run with a coverage measurement tool, and manually collecting and collating the coverage data from the various containers.
2. Do something very simple, like adding an extra non-parallel run of your entire suite with coverage turned on and only run it for the master branch.

Simple sound pretty good, doesn’t it?

Tests for the master branch are only run when you just merged something into it, like a PR, so the extra test run will not harm your development flow(we’re all working with PR these days and not developing straight on master, right?).

I personally don’t care that much about the change in coverage from a single commit. I am much more interested in the change of coverage from merging a full PR into master. What about you?

Here’s how the extra coverage step in the circle.yml could look for Python/Django:

- if [ $CIRCLE_BRANCH == ‘master’ ] ; then coverage run —      source=’.’ manage.py test ; coveralls ; fi