Towards Efficient Testing Practice at Ninja Van
The Ninja Van engineering practice encourages high-quality products, thus testing is a crucial part of what we do.
What Were We Doing
We were writing unit tests diligently for most of our microservices. Most of our services, data access objects, and helper classes have been tested. As we increase our number of tests, there are some unwanted effects from these tests.
Bugs At QA Environment
Correctness on Every Push Matters
In the past, we focused on delivering products as fast as we could. This practice was causing many bugs and crashes as the number of our microservices grew larger. One of the root cause is we do not test our code rigorously.
One of the solutions was to have dedicated testers/QA for every team. Some would probably ask “since we have QA, shouldn’t it be the QA that handles the burden?” No! All engineers should do their best to ensure the quality of their code BEFORE it reaches QA.
Testers/QAs are responsible for testing end-to-end business logic and generating edge cases toward that logic. If our code contain too many bugs, our time to deliver will be longer. This is because of the back and forth bugs reporting between the QAs and the engineers.
Slow Builds
Speed on Every Push Matters
One of our early practice in writing unit tests is to collect our mocks in a fat class (let’s call it BaseTest)and all unit tests class will extend that class. Since we keep adding classes and that BaseTest becomes fatter, running a test class could take up to 2s.
All engineers in Ninja Van currently have KPI to reach a certain code coverage target, and it could reach around ~80% (subjected to each team/service, because high code coverage won’t guarantee high quality). It might result in more than 800 unit tests (some services have a lot higher number of unit tests, but this is for my team case). These unit tests are run every time a code push happens, by our CI, so every bit of time saved, even milliseconds matters! Imagine if each test run in 2s, then if there are 800 unit tests a build will take up to 26 minutes!
Road to Improvements
Let Code Coverage Guide You
As part of our commitment to building high-quality products, our Engineering Excellence team helped other engineers establish tools like Sonarqube giving us code coverage targets for every library and servicse that we build.
Right Library for The Right Purpose
At Ninja Van, we use JUnit and Mockito for most of our unit tests. Additionally, we use H2 database to test our DAO (Data Access Objects) and Assertj for more fluent assertion.
There are some other libraries we use to run integration tests such as Wiremock and Testcontainers.
Wiremock has made mocking web services via mocking of HTTP servers easier, enabling us to mock internal service dependencies.
Testcontainers makes our integration with other software stacks, such as Kafka or Redis, easier. With a few lines of codes, you will be able to get those services up and connect to them with the help of docker. You can refer to the article that I’ve written in the past about Testcontainers.
Structuring Your Tests
Firstly, you could write all your tests as unit tests. However, once you reach hundreds of unit tests and you have set every push to run unit tests, you may notice that your CI starts to lag behind. One of the reasons behind this could very well be the library that you are using.
A library like Wiremock or Testcontainers needs a significant amount of time to boot up, especially Testcontainers. Testcontainers may need to download the Docker image and wait for the docker boot to finish before actually run the tests.
For this kind of behavior, you may realize that this should only run once for every test and you could do this by putting setup code behind @BeforeClassinstead of @Before(assuming you are using JUnit).
Another thing to consider is to separate your tests into unit tests and integration tests. Wiremock and Testcontainers could be more suitable for integration testing than a unit test because they provide more end to end functionalities. For more examples of how we do integration tests, please refer to this article.
Avoid Mock Static
Mocking static variables and functions is possible using a library like PowerMock. But that will come with a great expense.
This library will make your unit tests significantly slower. Typically our standard unit test class should run below 500ms per class. Using PowerMock runtime, at least in our experience, may require a boot-up time of about 1s, which doubles your unit test latency.
Avoiding static mocking will often lead to better design. It helps you to use your knowledges in object-oriented design patterns. For example, if you usually instantiate a class using the static methodAService.getInstance() , you may want to use the factory design pattern. Constructing new ServiceFactory().getAService() will enable a more flexible construction way and enable us to mock this getAService() method to return mocked AService .
# Wrap Static Methods in A Thin Class
One possible solution is to encapsulate those methods in a thin class, maybe even a singleton class. This way, you will be free to mock that singleton class.
Initialize Only What you Need for A Test Class
This is the last, and we think the most important thing, to pay attention to. We were using a class that has a collection of reusable mock classes, let’s call it BaseTest . It will make our development extremely easy, because then we could simply extend it, and reuse all the needed objects in that class, like Service1Testin the example.
However, what we didn’t realize was that when our tests began to grow, so did our time to test. One of our backend services has to run its unit tests for 6 minutes. Imagine if we have 10 commits in a day, and assume our CI runs those unit tests for every push, we could spend 1 hour just waiting for all the test runs to finish!
Now instead of doing that, there are some simple tricks to avoid this from start.
We have some suggestions to overcome this problem:
#Use Static Method to Create Commonly Use Mock Class
Using static methods will enforce us, engineers, to only call the methods whenever we need them. It will not be read by openMocks() and will not eagerly be initialized by it.
#Only List Needed Classes
This is quite obvious, but sometimes we fail to realize it until the very end. If you don’t want to end up refactoring all your test classes, then it’s better to follow this discipline from the beginning.
Results
We have decreased one of our service’s test running time by implementing these methods to all of the total unit tests.
As you can see, it has decreased from 6 minutes to 1 minute, an 83% improvement, and savings for each and every push we make!
We Are Hiring!
Are you passionate about building high-quality products? Come and join us! You are welcome to browse other things that we do in engineering, for example, the GAIA project that powers our flexible development environment and our EFK stack that processes ~3 Terrabytes / day.
Acknowledgments
This article was made possible through advice from Timothy Ong, Florian Hopf, Thung Han Hee. Thanks!
