Journey to Reduce PHPUnit Running Time

Unit testing is the process of testing individual components of a software application in isolation. By isolating and testing individual units, such as functions or methods, unit testing allows developers to identify and fix defects early in the development process, ensuring a robust and reliable final product.

Creating unit tests is a part of Mamitech development habit, we also set up CI (Continuous Integration) pipeline to run unit test on every new commit pushed to the code repository. As more and more features added, the source code also grows, let alone the number of test cases needed to be executed on unit test pipeline.

This article will tell about our journey to reduce unit test running time, and we use PHPUnit (Backend) for use case.

Parallel Testing

Back in around 2020, the number of our PHPUnit test cases already more than 3K. And it run for around 20–30 minutes long. It is unacceptable. Backend developer need to wait for half hour to see the result. If there were failures, the iteration got longer.

So we try to run unit test using ParaTest.

The objective of ParaTest is to support parallel testing in PHPUnit. Provided you have well-written PHPUnit tests, you can drop paratest in your project and start using it with no additional bootstrap or configurations!

Parallel Universe

But, it turns out not that easy. We found many errors about deadlock.

SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when try to get lock; try restarting transaction

That’s because we use same database for all paralleled test. Many test cases try to access the same table.

The solution is simple, actually. We just need to set up different database for each parallel process.

And it works! By paralleling unit test into 4 processes, our PHPUnit only running for no more than 10 minutes.

Utilize PHPUnit Test Suite and Run Them Separately

One year later, in 2021, the number of test case doubled and running time also back to 20–30 minutes long, even though we already use ParaTest.

At that time, we use default PHPUnit Test Suite. It means, we only run it on 1 CI pipeline. Here is what our phpunit.xml looks like:

<phpunit>
  <testsuites>
    <testsuite name="unittest">
      <directory suffix="Test.php">./tests/app</directory>
    </testsuite>
  </testsuites>
</phpunit>

So, let’s separate it into multiple Test Suites.

<phpunit>
  <testsuites>
    <testsuite name="unittest-1">
      <directory suffix="Test.php">./tests/app/FolderA</directory>
      <directory suffix="Test.php">./tests/app/FolderB</directory>
    </testsuite>
    <testsuite name="unittest-2">
      <directory suffix="Test.php">./tests/app/FolderC</directory>
      <directory suffix="Test.php">./tests/app/FolderD</directory>
    </testsuite>
    <testsuite name="unittest-3">
        <directory suffix="Test.php">./tests/app</directory>
        <exclude>./tests/app/FolderA</exclude>
        <exclude>./tests/app/FolderB</exclude>
        <exclude>./tests/app/FolderC</exclude>
        <exclude>./tests/app/FolderD</exclude>
    </testsuite>
  </testsuites>
</phpunit>

Then, we set up 3 CI pipeline to run each Test Suite simultaneously.

Running 3 PHPUnit Test Suites simultaneously

And it works!! Again, we can reduce overall running time into no more than 10 minutes long.

Now, another 1 year has passed since our last treatment. We have almost 10K test cases now, and overall PHPUnit running time becomes more than 10 minutes. Well, we can split them into more processes and our DevOps team already ensured the Autoscaling gitlab runner using AWS Spot instances, so we can reduce the pipeline running time again and again without big cost increase.

If you have more idea on how to speed it up, feel free to share your feedback in the comment.