How to effectively write Integration Tests with NodeJS+Knex+Postgres
Before going into detail, let’s take a step back and re-explore why we need integration tests in the first place.
As we write unit tests for our handlers, middlewares, functions or in short “units”, we enjoy the comforting boundaries of our service. There are lots of tools that can help us mock our dependencies effortlessly and 90% percent of the time it is still our code on the other side of the “mock” fence. We aim for the simplest units and try to achieve a high percentage of coverage. All in all, we feel comfortable adding/changing/removing code and deploying it on a Friday night.
But what happens when we reach the boundaries of our service and start integrating with other systems? After all, almost all applications persist some kind of state. At this stage we have limited options. We can still try to mock our integration with the help of testing libraries but those test will be flaky and will become stale at some point because we are unsure about the actual state.
Let’s look at a simple unit test example for our hypothetical “fetch all to-dos” service just to set the tone:
This test happily runs under the assumption of that database table still exists and its schema is still the same. But it is oblivious to any kind of change in our database. Even if you delete that table you’ll see the nice green “SUCCESSFUL” messages in your test output.
This might be just fine depending on your needs. If you are not developing on a system that is subject to change, why would you complicate things? But if it is a very dynamic system, I have some bad news for you.
But let’s brighten up the mood a bit, usually for integrations such as other HTTP servers are pretty straight forward and you can make some predictions about the possible outcomes, i.e if I get a 5XX response from an integration let’s return this “Please try again later!” to my client.
The real problem comes from integrations such as databases, message queues, Kafka topics — the hard persistence layers. You can still mock your “repository middlewares” (the integrations to your database), but your hands will tremble on a Friday night deploy and you’ll most likely leave it to Monday.
Things get even more hectic when you have an evolving persistent storage, such as ever increasing number of tables or table alterations or in short migrations. So what do we do then?
The first obvious solution is firing up your application and a database instance and manually go over these points one by one. For smaller codebases this might be a trivial thing, but for a busy project with multiple contributors it can ruin your plans for dinner.
Since this is the age of containerisation, the next solution could be to have a small docker-compose file that would orchestrate your database and run your tests without any mocking at all. This is how most applications are tested these days and it is really efficient. The only frustrating thing here is that it needs some degree of knowledge and expertise about how docker works or about manipulating that sneaky YAML file to achieve what you want. But wait, you wanted a simple way to test that piece of logic that inserts a database record right? So enter Testcontainers!
Testcontainers is a Java library that provides lightweight, throwaway instances of anything that can run in a Docker container.
Don’t mind the Java part, it also has a nodejs port.
The main advantage of using Testcontainers is being able to “code” your testing infrastructure. For example, how can one boot up a Postgres instance?
EASY RIGHT? Nah, that Postgres instance would actually suck. Let’s do it properly.
What we are doing above is basically creating a Postgres instance on a random port, exposing its 5432 port to the outside, and defining some access credentials including the database name. It is now ready to hot-wire to our application. When you run your integration tests, you can check the state of the created container if you’d like to with:
$ watch docker ps
This part is mostly dependent on your library of choice. For the sake of simplicity, I’ll assume you are using Knex. The next step would be to apply all our migrations on top of our newly created database.
We have tested our migrations even before the tests! Isn’t it wonderful?
Once we are done with our tests, we will release our resources. The easiest way to do it is hooking the logic to an automated process like afterAll or teardown section of your tests.
Let’s focus on actual tests. So we created a new Postgres instance, hot-wired it, and we can now seed our database and run our logic directly. Here’s an example test for an hypothetical “fetch all to-dos” service:
Looks more natural right? There’s no convoluted mock statements or dummy data to be refactored every time you change your business logic. It is also easier to read.
As with everything in life, this solution also has some drawbacks. Let’s finish this article with a quick pros and cons analysis.
- Increased visibility on potential problems that can’t be unit tested efficiently. e.g connection configurations, deadlocks, migrations etc.
- Using the same technology and version with your staging/production environments which increases confidence in the code you write.
- Super easy to integrate into CI/CD systems since all orchestrations are done through the code.
- Easier for on-boarding developers that have no prior docker knowledge to the project, as opposed to maintaining compose files.
- Possibilities are limitless. You can test your integrations with Redis, Kafka, Postgres, MySql or even your other custom made systems. Basically anything that can be Dockerised.
- Almost an order of magnitude slower in terms of test suite initialisation since it requires a fully fledged container to be booted up and be ready before running the actual tests.
- Hard to parallelise tests because of the cost of creating a container from scratch.
— Examples used in this article are available on my Github