Parallel testing for Laravel: How we cut test suite execution time to 1/3 in one day
RealBlocks is an Ethereum-powered digital trading platform for alternative asset classes. We make fund administration a breeze and facilitate liquidity through our secondary trading system. (We’re hiring for a Senior Front-End Engineer role!)
At RealBlocks, we use Laravel + TDD to help us write and refactor code quickly while maintaining quality. Last week the full suite took nearly 11 minutes to run on local, so we started looking at ways to improve our performance while maximizing efficiency and resources.
- Refactor the tests to avoid database/network?
- Use a (faster) in-memory SQLite database?
- Something else?
Refactor the tests to avoid database/network?
Oh no! Your tests aren’t isolated?!
Allowing tests to touch the database and network are considered bad practices for a few reasons.
- Tests run slower.
- Tests can fail for external reasons (e.g. third party API failure).
- It doesn’t discourage writing tightly-coupled code.
As a startup, our top priority is rapid iteration on our product. By writing non-isolated tests that touch the network, we can write new features faster, without having to write lots of mocks and stubs.
The trade-off for faster development in this case is slower test execution time and incurring technical debt on non-isolated tests that will require refactoring in the future.
We’re still a startup and only have ~ 300 test cases, so slower individual tests are fine (for now) as long we can run the full test suite in a few minutes. The problem is our suite is taking 11 minutes already!
In the most pure sense, the correct solution is to refactor the tests. But what if we don’t want to spend a two-week sprint (right now) refactoring tests for features that may not even make it to production? Can we delay this decision?
Let’s come back to this as a last option. We’re looking for a quick win to bring us down to ~ 5 mins execution time TODAY.
Use a (faster) in-memory SQLite database?
SQLite is fantastic lightweight database that is popular for testing since you can run it in-memory resulting in faster tear-downs and migrations. You’ll find Adam Wathan, Jeffrey Way, and others use it online in the context of Laravel TDD — it’s pretty damn fast.
One of the cons of SQLite, however, is that it doesn’t support all types of database migrations, so you’ll have to make sure that all your migration files are compatible with Sqlite. For example, if your migrations attempt to drop foreign keys, SQLite does not support this. There are workarounds, but we’d need to reword our migration syntax.
We didn’t plan on supporting SQLite when we got started, so to get SQLite running we’d need to rewrite a lot of our migrations. This also feels like quite a bit of work and may have future considerations — will we always need to be SQLite compatible? Let’s keep digging and see if we can find a better solution.
What if we could run the tests in parallel?
Paratest is a fantastic package that will run our test suite in parallel! After following the instructions in the readme, we can specify how many parallel runners you want to use with the -p option.
vendor/bin/paratest -p8Aaanndd… most of the tests fail. Why would parallel running cause our (green) tests to fail?
Parallel database migrations = bad
The first villain in this story is Laravel’s DatabaseMigrations trait. If you’re writing tests that touch the database with Laravel, odds are you’re using this utility to make sure your database is migrated before each test suite, giving you a fresh database to test against. This was super convenient before, but now that we’re running our tests in parallel, this utility causes tests to clash when handling the database.
While one test is migrating, another is tearing down, another is trying to seed data, and yet another is trying to insert data. Not good.
It’s almost like trying to run multiple experiments simultaneously using the same petri dish. One scientist is adding bacteria for their experiment while another is trying to measure their own results. It’s clear that this would no longer be a fair test. See cute dogs for further illustration.
Solution? Simply replace the DatabaseMigrations trait with the DatabaseTransactions trait. Instead of migrating and tearing down the database with each test, we will only migrate ONCE at the start of the test run. Each test will record the transactions it performs on the database, and then roll back those changes when it’s done executing. Neat!
Let’s run the tests again.
Better, but some tests are still broken…
Instant results! When run in parallel, our test suite now takes around 3 minutes instead of 11; that’s a huge boost for very little work on our part.
But we still have a few failing tests… Let’s take a look at the assertions and try to figure out why they work in sequence but not in parallel.
/** @test */function admin_can_view_transactions(){ $response = $this->actingAs($admin)->get('/api/transactions');
$response->assertStatus(200) ->assertJson([ 'data' => [[ 'id' => $transaction->id, 'asset_id' => $transaction->asset_id, 'quantity' => $transaction->quantity, 'price' => $transaction->price ]] ]);}
The issue here is that assertJson() is expecting a data array in the response, where the FIRST element in the data array is the transaction object described in the test. If we run this test on its own with a fresh database, it passes.
Since we’re running our tests in parallel, it’s possible for other tests to write to the same database that we’re checking against. There’s really no guarantee that our object will be the first element in the response.
Luckily, we can fix this by changing our assertion to something that’s guaranteed to be true.
$response->assertJsonFragment([ 'id' => $transaction->id, 'asset_id' => $transaction->asset_id, 'quantity' => $transaction->quantity, 'price' => $transaction->price]);
Now, even if there are other records in the database, our test should be able to find the transaction that we’ve specified in the json response.
Another assertion that created problems in parallel is assertJsonCount() .
/** @test */function admin_can_filter_transactions_by_status(){ $pending = $this->createTransaction(['status' => 'pending']);
$closed = $this->createTransaction(['status' => 'closed']); $response = $this->actingAs($admin)
->get('/api/transactions?status=pending); $response->assertJsonCount(1, 'data');}
When running parallel tests, the checks against json element counts fail because other elements have been inserted while the test was running.
Instead, we can refactor this using assertJsonMissing().
$response->assertJsonMissing([ 'id' => $transaction->id, 'asset_id' => $transaction->asset_id, 'quantity' => $transaction->quantity, 'price' => $transaction->price, ‘status’ => ‘closed’]);
After refactoring our tests to use DatabaseTransaction and guaranteed assertions that would hold up in a dynamic environment, our test suite is finally getting green while running in parallel.
Conclusion
Parallel testing is an awesome concept that exists in many testing frameworks and can provide a huge performance boost for developers doing TDD. Our team saw 3X returns from parallel testing using Paratest with very little time invested for performance optimization.
How does your team do TDD? Tell us about your favorite testing tips and tricks in the comments!
