How my unit tests went from 42 seconds to 1.8 seconds

TL;DR

As it turns out, the following is not a collection of revolutionary ideas about unit testing. If you already follow sound testing principles, you’re most of the way there. After you’ve written those tests to be as fast as they can, making sure they all run in parallel will make you feel so good inside that you might just write a blog post to share your joy with the world.

My approach to unit testing

Over the years, I’ve experimented with just about every approach to writing and running unit tests. The paradigm I’ve settled on for web services is something like the following:

// Some route handler or "controller"
$router->get('/api/items', function() {
$itemsService = ItemsService::getInstance();
return $itemsService->fetch();
});
// Service layer
class ItemsService extends Service
{
/** @var ItemsProvider $provider */
private $provider;
   public function __construct(Provider $provider)
{
$this->provider = $provider;
}
   public function fetch(): array
{
return $this->provider->fetch();
}
}
// Provider layer
class ItemsProvider extends Provider
{
/** @var DbInterface $db */
private $db;
   public function __construct(DbInterface $db)
{
$this->db = $db;
}
   public function fetch(): array
{
return $this->db->select()
->from('items')
->query()
->fetchAll();
}
}

There are three layers in this lasagna:

  • Controller — this is the class method that handles an HTTP request. Typically, the controller will clean up user input and send those along to the next layer, where the business logic takes place. Ideally the service object is injected into the controller, but this is a subject for another day.
  • Service — this layer is meant to receive data from some Provider object, possibly process this data in any way, then return it to whoever called it.
  • Provider — this layer does the direct interaction with the true source of the data (a database, an external API, etc.).

Often times the service layer will have some “passthrough” methods, where it’ll call the appropriate method on the provider object, and return that data without doing anything to it. This may seem wasteful, but that’s intentional.

The main purpose for the provider layer to exist (for the purpose of this post) is so that at test time, there is at most (most of the time, anyway) one method to mock. For example, if the service method receives some data from the provider, and manipulates it before returning it, we can test it in isolation as follows:

/**
* @test
*/
public function testingSomeServiceMethod()
{
$mockProvider = mock(ItermsProvider::class)
->method('fetch')
->returns([1, 2, 3]);
   $service = new ItemsService($mockProvider);
   $out = $service->mult(2);
   self::assertTrue($out[0], 2);
}

In other words, at test time, if you don’t want to have the provider object actually touch the database (or make an actual external API call, or send an actual email, etc.), you can mock the provider object and manipulate of one of its methods returns. In contrast, if all of the database logic was in the service layer (or in the same layer that uses that data), then you’d have to mock a lot of methods.

Running PHPUnit was taking 42 seconds

The current project I’ve been working on uses Lumen (Laravel), which has an interesting approach to testing interactions with the database. It uses migrations, which I had not worked with before, but now I can’t imagine working on a new project and not using it.

The gist of it is that with migrations, you programmatically describe your database schema, then at run time, the database schema can be created and destroyed for each test method. This way you can insert and delete data into a real database, and assert on whatever side effects you like. To make things easier, at test time, you can also use Sqlite and create this database in memory. So you’re testing against simple, throwaway database that has the same structure as your production one.

Of course, although this might be faster than setting up and destroying the same database you’re using in production, it would still be slower than not interacting with the database at all. At some point, however, you will want to make the effort and have your unit tests touch the database layer directly (but this, too, is a topic for a different conversation).

After my boss pointed out to me that taking power naps while we ran PHPUnit was not acceptable, I decided to look into what was going on.

It took a bit of digging, but the first change we made was actually pretty obvious. As it turns out, another feature of Lumen that goes along with migrations is seeding, which is an easy way to automate the seeding of your database with random data. This is particularly useful during tests, where you don’t really care for the contents of the data, but simply that there are several records in the database that you can test against.

The issue with this was that our seeding scripts were generating 50–100 records per table. As it tuns out, configuring it to generate 5 or 10 records per table was enough for all of our existing tests.

Optimization #1: don’t seed your database with more data than you need

As I mentioned, this is an obvious thing, but not paying attention to it is easy to justify. Since both the migrations and seedings were executed before each test method that touched the database, reducing the amount of records generated by the seeding scripts gave us the first performance boost:

Optimization #2: don’t use XDebug unless you need it

Again, this is another obvious bottleneck, but one that’s easy to miss. A while back I started to use breakpoints in PHPStorm, so I enabled XDebug. Turns out that for the past few months, I had not remembered that it was still enabled, so every time I ran PHPUnit, it was running with XDebug turned on. So, unless you’re setting breakpoints in your code or generating test coverage, or for some other reason you are using any of the features of XDebug, you should consider turning it off. The gains here were also significant:

I really should have been more scientific here, and should have put the seeding configs back to what they were before, then reran my test suite with XDebug disabled. This way I could see what impact XDebug and the excessive seeding had individually. Still, making both of these adjustments now made running PHPUnit acceptable again.

Optimization #3: don’t make network calls inside a unit test

Another obvious concept, but one that you may miss when there are a lot of tests in your project. The way I noticed this was even happening in our test suite was by looking at the progress bar as PHPUnit ran. Two tests took particularly long to finish. I then disabled network, and those tests now took a long time finish. As it turns out, there were two tests that were actually calling a third party API. The solution was to mock the data provider for the particular service under test, and the tests now ran in a few milliseconds (actually, I didn’t measure how long those tests took, but it was visibly faster).

Also, I don’t remember when I took those two tests out, so I don’t have a screenshot of my PHPUnit results before/after this change. Furthermore, since there were only two of tests making calls across the network, the effect wasn’t too drastic.

Optimization #4: don’t run your unit tests sequentially

Concretely, be sure that your test suite ran all of your unit tests in parallel. There are lots of tools that do this in PHPUnit, but the one I settled with was Fastest. Setting it was was trivial (on PHP 7.2), which was refreshing, since this was the third tool I experimented with in order to run PHPUnit in parallel.

Now we even commit code without using

git commit -n -am 'Don'\''t got a minute to wait to commit this'

to avoid running the unit tests for 41 seconds on a pre-commit git hook. This is particularly aggravating when you want to commit code, wait 38 seconds, and one of the tests fails. Again, as the topic for another post, I might talk about how we addressed this effectively. However, now that running the entire suite takes less than 2 seconds, those workarounds are almost completely irrelevant.

Bonus: run your code coverage in parallel

Since we run our unit tests (and test coverage generation) in Travis-ci and Coveralls, I thought I’d try running that in parallel as well. Our build time went for 3 minutes to 2 minutes, which I thought was pretty handy. The problem was that the test coverage was all messed up. Looks like if you want to generate test coverage when using Fastest, you need to set up another step so that the report from each parallel process is combined into one. This seems pretty straight forward (topic for another post?), but I didn’t want to take the time to implement it right now. I thought that just improving the unit test experience during development was good enough for one day.

Like what you read? Give Rodrigo Silveira a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.