Isolating JUnit Tests for Fun & Profit

Continuous Integration’s the name of the game!

Where we’re coming from

Testing our software is fundamental to Personal Capital’s culture of product development.

We lean heavily on Jenkins for continuously integrating our engineering teams’ changes to our front and backend software.

On the backend, we use Java’s JUnit + maven-surefire plugin to run a suite of automated tests which verify that our APIs behave as we expect them to.

Things generally go smoothly however there are some tests we have that attempt to test large swaths of our software at once.

These tests often break when small changes are made in our codebase, or worse: they only pass some of the time, making it very difficult to identify sources of uncertainty in our software.

Where we’re headed

In order to expand our CI/CD process, we’re requiring pull requests to pass tests before they can be integrated into our shared codebase.

To accomplish this we need to:

• eliminate unreliable tests like mentioned above, and
• cut down the time it takes to run our tests — slow tests are tests that no one runs!

It’s critical for technical and non-technical team members to trust our tests. This means holding new and refactored code to the same high standard of requiring tests to run and pass before any additions or modifications are allowed to be made to the codebase.

In order to accomplish the two goals above, we can use JUnit’s @Category annotation to tag slow test classes and methods. Once tests have been tagged, we can either include or exclude them from our test runs.

This meant creating a simple interface, for example:

package com.personalcapital;
public interface SlowTest {}

This interface can be used in combination with the @Category annotation to identify slow test classes:

import com.personalcapital.SlowTest;
@Category({SlowTest})
public class FooTest {
    …
}

Or slow methods:

public class BarTest
{
    @Test
    @Category({SlowTest})
    public void testSomethingThatShouldBeMocked() { … }
}

Invoking the test Maven goal, we can run only the slow tests, e.g.:

$ mvn test -Dgroups=”com.personalcapital.SlowTest”

Or exclude all slow tests:

$ mvn test -DexcludeGroups=”com.personalcapital.SlowTest”

We can exclude slow tests when automatically running tests for a proposed change, and run our slow tests later on when we evaluating a manual release to production.

Using Jenkins’ JUnit Plugin, it becomes very easy to identify slow tests by treating your test output as structured data:

What to look forward to

We’re delivering tools for our customers’ to better manage their personal financial lives at a faster rate than ever before, with a larger team than ever before.

We’re striving for the holy grail: continuous delivery, where a team member might join Personal Capital and confidently contribute code to production within hours of walking in the front door.

Come and help us reach that goal!