Running Parallel Functional Tests
How often have you just found yourself sitting and waiting for some automated tests to finish?
How many times have you asked yourself “is all this waiting worth my time?”?
As a software engineer at Fiverr, I have had to meet many tight deadlines without compromising on the quality of my code. In this article, I will show you how, with a little extra effort, you can make your tests run faster, saving a lot of time but maintaining high quality standards for your code. At Fiverr, we have saved up to 50% of the time it takes to run tests.
Why Is it a Problem?
Depending on your organization’s philosophy, you might find yourself writing many different types of tests: unit tests, integration tests, regression tests, etc.
At Fiverr, we are primarily focused on unit tests and integration tests.
Some tests are written for stateless flows — given a certain input there is a specific expected output. These tests are easily run in parallel since they do not share a common resource, such as a database. But there are many flows that are stateful, and testing them concurrently using the same resources, often the same database, may cause the tests to fail even if they should logically pass.
Additionally, it is common to instruct each test to reset the state (truncate the database) before or after it is done in order to avoid leakage.
To avoid these issues, it is common to run these tests sequentially, making your pipeline run slower and slower with each added test.
In this article, I will focus on how to run the stateful flows tests simultaneously.
Ensuring Test Isolation
While there are many popular frameworks that provide some sort of API for parallel testing, some of them, like Gradle (shown below), also specify their limits:
“When using parallel test execution, make sure your tests are properly isolated from one another. Tests that interact with the filesystem are particularly prone to conflict, causing intermittent test failures.
Simply put, if we can make sure the tests are isolated, we can run them concurrently!
There are several ways to successfully reach test isolation:
- Divide the tests into groups such that there are no shared resources between any tests in different groups. This process will allow you to run all of the test groups in parallel, while, within a group, the tests run sequentially.
Nevertheless, this approach can be very difficult to maintain, and, eventually, you might end up with running all of your tests sequentially. - Give each test its own resources (i.e. each test gets its own Mongo collection, Sql table, etc). However, you need to keep in mind that there can be a limit to the amount of resources the test environment can hold. You will also want to make sure that, when limiting the resources, the tests still effectively reflect the behavior of the production code.
- Use dependency injection throughout the codebase to inject each model with its resource upon setup. Doing so means that every flow will need to be resource-aware and declare which resources to utilize. This solution might fit some cases, but it also has drawbacks which will be mentioned later.
At Fiverr, we decided to explore the second option.
Context-Based Database Connections
To give each test its own storage-space, we need the following:
- A single module, or wrapper, responsible for DB communication (for each database we use for testing).
- A way for the DB connectors to identify which test is running in order to contextualize the test.
We also want the information to be transparent to anyone who may write tests in the future.
Luckily for us, it is a common practice to have a wrapper for any third-party package.
Here is a simple example of a MongoDB wrapper, written in Go, using the official Mongo-Go-Driver. This wrapper exposes the API for a specific collection.
And below is the same for Redis:
Meeting the second requirement, however, is a bit more complicated. We can either identify the current test before running it or identify it during runtime.
Identifying the test during runtime allows us to verify that our solution will also work on integration tests where shared resource usage might be implicit.
Identifying the test before running allows us to implement the third option: injection. Still, there are some important considerations to take into account:
We want our solution to be transparent to future developers who will write and run tests — there is no need to implement the flow in any specific way to be able to test it in parallel to other tests.
We already have full test coverage, and re-writing all of our flows and tests to allow injection would be very costly.
Considering all of the above information, we have come to the conclusion that the connectors need to identify the current test during runtime.
Searching for Test Context
Initially, we thought searching for test context would not be a problem: we can look at the stack-trace of each running test and at some point be able to see the test’s name (since it is a function, and the stack-trace includes function names).
We quickly learned that stack-traces are nice when you have a single process in charge of your flow. But we want to test cases that involve more than one process and cases that use non-blocking commands like Go’s goroutines.
It turns out that goroutines do not keep their parents’ traces as if they were their own. Instead, their traces start from the call to the goroutine (the go command).
The Answer
The tests are run by a Runner. It can be a command in the terminal, or it can be the IDE or another app. This runner considers the test files as the executable, and most languages force you to use a convention for naming these files (XXXX_test, test_XXXX, XXXX_spec etc.). All you need to do is just ask the operating system the following:
Tying it together:
First, we will need to update our application setup to allow for configuration of our connections. Since every database may require a unique approach to use the context, we leave it to the wrappers to get a ContextNameEnricher from the app’s configuration object and employ it properly.
It is worth mentioning here that this setup is also necessary when running integration tests in order to set up the connections.
The following is a shallow example including only the ContextNameEnricher option:
Note that, by default, the ContextNameEnricher does not change the received context string.
On our main package, we have no change:
Our connector wrappers are updated to use the ContextNameEnricher from the app configuration.
Mongo
Mongo will use the ContextEnricher to manipulate which Collection it connects. For example, users collection with the default enricher will remain users, but it will change into <context_name>_users when using a non-default ContextEnricher:
Redis
For Redis, we did something a little different. Since we do not have a “collection” in Redis, we added this test-name-prefix to the keys:
Tests Setup
Finally, the last step is to update our tests setup to use the ContextEnricher option and configure the app to use test-name as the context enricher:
Results
At Fiverr, testing is part of our CI/CD process. You cannot deploy any app before it passes the tests. After adding the ability to run these tests in parallel, the testing part of the CI decreased from taking approximately 5 minutes to 2.5 minutes — a 50% reduction for every process!
This addition, our infrastructure not only saved us a lot of time in the context of the project we worked on , but it will also make future projects much quicker and smoother. Before this change, every test we added to the project would make the CI process take longer. Now, adding a test does not necessarily add time — testing time is only affected by the longest test and the amount of parallelism allowed by the machine running the tests.
In conclusion, investing a bit of extra time to expedite the process is extremely helpful and will pay off in the long run.
Fiverr is hiring, learn more about us here
