Optimizing unit tests with shared context

In my years of experience as a software engineer, I’ve noticed that in some projects, the CI (Continuous Integration) and CD (Continuous Deployment) time tended to grow exponentially with the project’s growth.

The reason seems obvious: once a project grows, it tends to have more code to test, tests, or even more steps in its CI/CD. But this is a vague and unquantifiable answer that didn’t satisfy my curiosity.

Something else I observed was that in several of these situations, unit/integrated testing was the most costly step in the entire process. One day, in a relaxed conversation between coworkers, I mentioned this, and one of my colleagues suggested that I use a shared context between my tests, especially those that accessed a database, whether real or in memory.

I did so, and, to my surprise, we had a significant gain in time when executing CI/CD on that project. Since then, I have had the standard of using IClassFixture (a shared context) in all my xUnit projects that access the database. Still, I didn’t have any hard data to support this decision until I realized the following…

If you use an in-memory database but do not have a shared context, you will see below that you can reduce the execution time of your tests by up to 4.44x by applying the shared context (and using less than a quarter of the time it currently takes).

The result is even more surprising if you use a temporary database without a shared context in your tests. The running time can be reduced by up to an impressive 96.76x — meaning you’re using less than 5% of the running time of your tests.

Using a temporary database with a shared context is even more efficient than having an in-memory database with a shared context. To be more precise, it is up to 6.22x faster.

And like practically everything in life, tests with shared context aren’t only full of positives. If you want to use them, one of the drawbacks you will face is that objects inserted into the database by a test can interfere with another test that is not even in the same class.

This occurs because the database is now shared along with the context. If a particular test depends on creating an entity with a specific key, but another test has already created it, you will see a test fail due to an insertion attempt of an already inserted primary key.

Therefore, remember to consider your tests as a whole and not as isolated tests.

Without further ado, these are the results I came to present to you today! But wait a minute, because without showing the base data from my study, this is all just fancy words. If you’re curious about how I found these results, grab a coffee and come with me to the development chapter of this article.

I understand that science doesn’t have all the answers to our questions and that the scientific method can continuously be improved, but it is still the most efficient way to test a hypothesis.

So, I had two hypotheses before carrying out this study:

1. Using an in-memory database with shared context was the most efficient combination for tests involving databases
2. Tests tend to lose performance and take longer to run as migrations are added to the database

I was surprised by the result because my first hypothesis was wrong. A temporary database with shared context is more efficient than an in-memory database with shared context. But I wasn’t entirely wrong because my second hypothesis is correct.

I wrote a project in .NET Core 7 and Entity Framework Core to test my hypotheses and coded some repositories and entities to try with xUnit. You can check out this project by clicking here.

With the initial project ready, I coded 15 different tests for the repositories created to run these 15 tests in 4 different situations:

1. Using an in-memory database without IClassFixture;
2. Using an in-memory database with IClassFixture;
3. Using a temporary database without using IClassFixture; and
4. Using a temporary database with IClassFixture.

With the temporary database, I’m referring to a temporary Postgres database, which is created at the beginning of tests and then deleted. The database was built locally, so the latency of a connection to an external server was not considered.

But let’s face it: latency is irrelevant when accessed internally in a cloud.

The tests were carried out only with Postgres. Other databases were not considered since the objective is to compare the efficiency of using the shared context, not which database is faster.

Screenshot of Visual Studio comparing tests using a temporary database. On the left, the tests are not using a shared context, and a new database is created for each test class. Unlike this, on the right, the tests use shared context, and only one database is built for all tests that are part of this test collection.

Apart from minor modifications inherent to how the single/shared context is coded, the tests are identical. Same arrange, same act/method to be tested, same expected assert/result.

To make the study result auditable, I ran the tests at the prompt with the following command:

dotnet test --verbosity n --logger:"trx;LogFileName=v{version}\{file-name}.trx"

This way, it was possible to generate a report with the execution time in milliseconds of each of the tests. Are you curious to see these reports? You can find them here, next to the project repository.

So far, I had 20 reports from 60 tests. What interests us in these reports is the average execution duration in each scenario to compare them.

I was also concerned about categorizing, summing, and organizing the results in a way that is not error-prone. After all, there are 1200 test executions in which an incorrect millisecond can generate an erroneous result reading. So, I coded a result collector that did this in a standardized way.

You can see its code here (I ignored some good practices in this project; I tried to make it as simple as possible).

Screenshot of the Windows console after running the tests via the command line.

With the compiled results, it can be seen that tests using an in-memory database and shared context took an average of 47.58 milliseconds, while the same in-memory tests without the shared context took an average of 211.17 milliseconds.

The same tests, but using a temporary database and shared context, took an average of 7.65 milliseconds. Removing the shared context for the temporary database, this time increased to an average of 740.29 milliseconds.

Test types, namely group test results: in-memory with a shared context, in-memory without shared context, temporary database with shared context, and temporary database without shared context. The average results were 47.58 milliseconds, 211.17 milliseconds, 7.65 milliseconds, and 740.29 milliseconds. Both tests with shared context proved to be more efficient than tests not using shared context.

Before we move on to the conclusion, I have one more result to share here. Remember I said I was correct in my second hypothesis (tests lose performance as migrations increase)?

To do this, I added 15 migrations to the project, simulating the organic growth of the database. See them here. These are common migrations. Add or remove columns, add or remove entities and foreign keys, and change column name or nullability.

As the objective is to evaluate whether there is a loss of general performance in the tests, I focused on something other than the migration content. However, the complexity of the migration affects its execution time. But, this is a hypothesis to be tested in another study.

Still, I chose to have some content in the migrations instead of creating empty migrations so that the bank’s growth looked as organic as possible. With the migrations created, I generated another collection of 20 reports, collecting the results the same way as previously described.

Here, you can see the reports from this second round of tests.

Test results after simulating database growth by inserting 15 migrations. A loss of performance is observed in all types of tests. In-memory tests without shared context had the most minor performance loss, with an increase in execution time of just 3.73%. Next comes in-memory tests with shared context, with a 6% increase in execution time. Then, tests on a temporary database with shared context, with a rise of 10.32%, and tests on a temporary database without shared context, which had the most significant loss in performance, increased their execution time by 49.33%.

Using a temporary database with a shared context proved to be the most efficient way of using a database in unit testing. However, the in-memory database with shared context suffered a minor performance loss more than the temporary database with shared context after being subjected to simulated growth (6% and 10.32%, respectively).

And there is one more difference between the temporary database and the in-memory one. A careful eye may have noticed that the Order repository tests with an in-memory database did not break due to the lack of Customer entity (here and here). However, when using the temporary database, it was necessary to create the Customer entity previously (here and here).

In other words, if you want behavior closer to reality, you should refrain from using the in-memory database since it will not fail when a foreign key is non-existent. If that’s not important for your tests, that will be fine.

It is also possible to categorically state that using a temporary database without a shared context can become a time bomb for your project since a 49.33% performance loss was observed in this category after the database was subjected to growth simulation. And your future self will not like having to deal with this problem.

Writing is something I enjoy, but every article needs to have an end — and we’ve reached that end here. There is room for more research, as I still want to find out how the content of migrations affects test performance and whether the performance loss is arithmetic or geometric. I also want to write an article detailing how to configure shared context properly and avoid the pitfalls. Follow me so you can see the next results!

Good knowledge is shared knowledge ✨

See you =D

PS: This article is also available in Portuguese here.

Excited to be part of our global mission to empower hospitality businesses? Explore our career opportunities on our dedicated page!