Python Integration Tests: docker-compose vs testcontainers

Developing a new feature in a growing startup can be a real challenge, especially when you begin to work as a software developer in that company. In this article, I will try to share the last experience that I had, which involved a lot of principles like TDD, unit tests, and integration tests. Let’s dive in.

What is an integration test?

There is the famous pyramid of software testing, that goes from unit test to integrated (or End to End, “E2E”) test, where the middle step is called “Integration test”.

By the way, at Spotify, they changed this traditional approach by replacing the pyramid with a hexagonal shape (“Honeycomb”) which takes the focuses on integration tests.

The traditional testing pyramid: credit to https://engineering.atspotify.com/

The honeycomb testing credit to https://engineering.atspotify.com/

But what is an integration test and why do we need it? I won’t take a long on that definition, there is a ton of literature on the web that will answer far better than me… but let’s take an example to illustrate the difference between those two types of tests.

Let’s take two functions, one for inserting a new row in a database and the other that will call it :

We have an insert function that could be a part of a repository. It will insert a new row in the database session and return the id of the newly inserted row.

And we have a create_new_item function that will create a new item, add a value to one of its attributes and call the insert function.

For testing the behavior of these two functions we have two ways:

1 ) Unit tests

We achieve this by testing only the create_new_item function (since the second one will be a part of the integration test).

Here we simply create a new item, giving it a name by passing it as a parameter to our create_new_item function. We assert that the returned item has the same name property as what we passed to the create_new_item function.

2) Integration test

The previous unit test only focused on item creation as an instance of the Item class. But it did not test the actual row creation in the database: maybe we have some SQL CONSTRAINTS there, Foreign Key, type mismatch, and so on…that we need to test.

We certainly could have a fake repository that will test the insertion of a new item. That will take our unit test to the next level …But this still will be considered as a unit test.

If we have a real database instance with a defined schema, and we write to it within a test, well this is an integration test. Because it will test our application behavior when it communicates to a third-party component of the architecture. This third part will be an integrated part of the production.

Let’s code.

Setting up our environment

The two solutions that we will discuss in this article are containerized solutions. In other words, you will need Docker in both of them. So make sure you have docker installed on your system (and if not, and you don’t know how to install it and what it is…).

For the first solution, you will need “docker-compose” (even though we will see that we can use plain docker for that solution, compose is always a far most easier way for all commands that are more than “docker run <IMAGE_NAME>”

For the second solution, you will need the testcontainers library for python and it will be a part of your project’s dependencies (the “requirement.txt” or “pipfile” or whatever dependency tool you prefer).

And no need to be precise that you will need Python! I use the 3.9 version here but 3.5+ should work too.

We also need the Pytest library which will be used as our test framework (instead of the built-in unittest library, mainly because of the fixtures feature it provides).

Once all this is installed: let’s do it!

A quick Code Review: What we are testing! (Optional but recommended)

If you want to focus on the core of this article (comparison between docker-compose and testcontainers library) you can skip this part and jump to the next where we begin with the docker-compose solution.

All the code for this article can be found HERE. This repository has 2 branches, one for the docker-compose example and the other one for the testcontainers example

Here is our entry point to the app. We have the setup_app function that will call a Metadata instance from the SQL Alchemy library.

This one will also need a database engine.

Let’s break down the Metadata and the Engine objects.

The Metadata object will contain all the schemas declaration and it has the main advantage of not being tightly coupled to the ORM.

Here is our metadata declaration.

We declare our schema as a Table object and attach it to the Metadata instance.

Now let’s declare an Engine.

We use the create_engine function from SQL Alchemy. This one needs a database URL.

The “build_url” function is very important for understanding the core of this article because it will be a central point when we will discuss the difference between the two testing approach that we are comparing.

The Configuration class bellow is a simple Singleton instance with a get method. The get method will retrieve elements from a configuration object, loaded from a simple JSON file.

The SingletonMeta is out of the scope of this article but you can still look at it in the source code. It basically ensures that we refer to the same memory address each time that this class is called…

Now let’s focus on the core business: create a new row in the database.

First, we will create our insert method in our repository.

Our insert function receives a name and a session object.

We then call the session’s execute method passing the generated insert query.

Finally, we return the lastrowid attribute of the newly created item: this new attribute will not be registered until we call the commit method of the session. Following the principle of the repository pattern, it is indeed the responsibility of the caller to perform the commit action.

We will then create a pretty straightforward service function as we did in the introduction.

We open the Session object with our database engine. We call the insert function from the repository. We commit and finally return the new row id.

We are finally ready to write the integration test! For sure we should have written some unit tests before, but again we want to keep the focus of this article on the integration testing.

How to perform an integration test, the docker-compose way

Let’s write our first integration test. We will insert a row into the database and make sure that it was inserted by retrieving it.

For this part, we will take advantage of pytest fixtures.

A fixture is something that you need for preparing your test (the “Arrange” step) and is not directly what you are testing. It kind of replaces the setUp and tearDown function at once.

Let’s first create some fixtures to set up a database engine that will connect to our actual database and a session that will communicate with it (I took this code example from here but it is pretty straightforward):

We have three fixtures here. One that creates the database engine based on our build_url function. A second that will create a scoped session based on that engine. And last but not least, a fixture that will return (yield !!: allowing cleanup actions like the rollback after the test suite is complete) the session.

Now that we have our fixtures ready we can use them in our tests. Pytest has a kind of dependency injection based on the name. Just by adding a parameter with the same name of our fixture, Pytest will inject it.

First, we “markerize” our test with “pytest.mark”. Markers are another very cool feature from pytest. It allows us to tag our test and then run the test suites by selecting some tests.

You can add whatever name you want after the “mark” and Pytest will register them. Now we can select only integration tests suites by running the following command :

pytest -m "integration"

And let’s say that you add another mark like “create_items”. You will be able to select it with that command :

pytest -m "integration and create_items"

How cool is that ?!!!

By the way, every custom marker that you add should be registered and explained in a “pytest.ini” file, which could be :

[pytest]
markers =    
    integration : run all integration tests
    create_items : tests related to the items creations

And now you get your test more and more documented!

Okay, let’s focus on the other lines.

So we are using the “db_session” fixture. Next, we call the create_item function of the item service. It should insert the row in the database.

We then use the db_session for retrieving the newly inserted row and assert that it has the same name as given.

Let’s run that :

pytest -m "integration"

Oh god…why our database is not connected … well because we don’t have a database yet!

But the question is: HOW DO I GET A TEST DB ??

Yes because if you read these lines it’s all about how getting external components such as a database to test our application in a “real” environment.

Well, Bob will say (I have nothing against this name): “Nothing too complicated here…I will install MySQL on my computer, and I will write in my connection string into my config file and run my test”.
Yes. It’s okay. Nothing wrong with that. But what if you need a more complex application as a third party? and what about the CI pipeline? (Continuous integration pipelines that run on the cloud-like GitHub actions…you have no database there !).

No, definitely it’s not the ideal way. In my previous job there was this huge “Test DB server” that everybody uses as a component for integration testing…but guess what: it was overwhelmed by garbage and anybody wanted to take the responsibility to clean it up since smart developers used it in production (No I am not joking: totally true).

Docker to the rescue!

We can certainly run a plain docker container by typing this command :

docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=items -p 3306:3306 -d mysql:tag

But this is not really maintainable, right? What if you want to add a volume to save your data?

docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=items -p 3306:3306 -v db_data:/var/lib/mysql -d mysql:tag

What if this port is allowed? Okay, you get the point…changing command is certainly not the optimal way. So let’s take a look at the docker-compose file.“

Much more clear (for me at least). We run a MySQL image as a service called “DB”. We give some environment variables and we map the ports…everything looks good.

Let’s run

docker-compose up

And now let’s run

pytest -m "integration"

Yeah, we run our first integration test! That’s cool.

There is some issues/challenges with that approach though :

1- You are now dependent on the docker-compose file. Again you could always run a plain docker command but as we saw, it’s not easy to maintain.

2- You must check that what you put on your environment variable is the same configuration that what you put in your configuration file. There are a lot of smart solutions for that though, but with the simple solution that we provided here this is not the case and you have a duplicate source of configuration.

3-You need to run the docker-compose command before running your tests. You cannot just install your application and run “pytest”. It won’t work since you have no database instance.

For all those points, testcontainers library is your friend. Let’s dive into it.

Testcontainers solution: a built-in solution for running integration tests.

Testcontainers is a library initially wrote in Java. The principle is to encapsulate the basic docker commands in a code API, to manage containers directly from your test suites.

Behind the scene, this library is using UNIX sockets to send directives to docker but we will see that in a few minutes.

The good news is that there is a port of that library to python.

They provide an apparently very simple example

Okay, let’s give it a shot within a pytest fixture.

Here we call the fixture with an “autouse” parameter to run this fixture before all others tests.

So hopefully, no need to “docker-compose” anymore…just run

pytest -m "integration"

Ok. It is not working apparently.

The first thing we can check: does this command runs at least a MySQL container?

Let’s open a double window terminal and check that by running a watch on the docker ps command

first terminal

watch -n0.1 docker ps 

second terminal

pytest -m "integration"

No running containers here…

The truth is out there…

Let’s see our traceback :

Okay. All begin with that “from app.app import app”. Python will always first resolve the imports before running anything else. And running means also executing the file. And if you recall we have this in our server.py :

Yes. By importing the app, we basically called

metadata.create_all(engine)

Which will need a database engine to perform the migration from the metadata to actual tables in the DB. It expects a URL that works (that connects to a real DB instance). But you guessed it: we still do not have any running database at this point since the testcontainers fixture was not called yet!

Let’s just remove this import to see if it changes something.

Yeah !! We have a running container!

But …

Why our tests are failing again ?!

Well if we have a look at the docker ps output we can see that the port of the new container is not the default port that we expected, the one that we put in our configuration file (3306) :

And if you start again, you may notice a different port at each run!

Let’s take a look at the testcontainers API to see if we can bind the new container (or at least expose it) to the port that we want.

For that we can go to read the doc…or jump into the source code of the library (my preferred way), to understand better what’s going on behind the scene of this context manager :

with MySqlContainer(MYSQL_DATABASE=db_config.get('database')) as mysql:
    yield mysql

Press Ctrl+B in Pycharm,

F12 in VSCode

Vi, nano, vim … to “<YOUR PYTHON ENV>/lib/python<PYTHON VERSION>/site-packages/testcontainers/mysql.py”

And you get there :

As we can see here, there is no real option for configuring the port…and now if we have a look at the documentation, we understand that this is on purpose.

From the docs :

From the host’s perspective Testcontainers actually exposes this on a random free port. This is by design, to avoid port collisions that may arise with locally running software or in between parallel test runs.

Ok. So we need to find a way to change the configuration object to match this new port automatically generated by testcontainers.

What could use the provided method : “get_connection_url()”.

So in the db_engine fixture, we could replace our build_url function with the URL we get from testcontainers.

For that, we could inject the mysql_instance fixture as a parameter of this db_engine fixture. Let’s give it a shot :

so we move from :

To

Mmmm but the test still failing

If you recall, 5 minutes ago we removed the create_app function which was responsible for calling the metadata migration. SO there is a DB….that’s good. But there are no tables!

No problem…let’s create a new fixture to perform the migration for us (SPOILER: very bad idea).

We import the metadata object and now we perform the migration with the db_engine fixture.

Let’s run the test…

Hmmm. Why? Well, this is because, in the insert function, we call the engine…but not the same. If you recall :

We import the engine from our DB folder which is dependent on the “build_url” function.

Well, we could decouple the create_engine call and let it receive the URL as a parameter…but then we would have to patch this call in the test for injecting the brand new URL we received from testcontainers…OK! That’s not what we want. I told you…this was a bad idea.

What we could modify the configuration object and inject the new configuration we get from testcontainers there.

So instead of performing the migration in the test (bad idea…really), we keep the migration at the application responsibility level.

We will create a simple function that will change the configuration of the DB. Since our Configuration class is a singleton, each time a function will call it, it will return the same configuration (and will not init it from the configuration file). So if we change it, it will change it “everywhere”!

Let’s give this a shot :

So we update our configuration according to what testcontainers “decided” for us. Let’s run our test!

Oh, come on !!! Why?

Well, this is because of our db_engine file. Look at it a minute again… :

And from our conftest.py we import the “build_url” in order to yield it as a fixture… a bad idea! Here will be the flow :

import db_engine. =>

execute “engine=create_engine(build_url())” =>

call Configuration Singleton=>

get URL (based on not updated config) => bind the wrong URL to the engine.

Because again, the python interpreter resolves imports statements before all …so the file is executed and the engine gets its URL from the Configuration class before it was updated!

And every other call will return the same engine bounded to the first URL (not the updated one).

So we have to encapsulate the engine in a function instead (and replace every call to that function). For sure there are better ways, we could create a smart class that manages engines for us and avoid the unneeded calls to that function by checking a kind of registry by DB URL… but you get the idea.

And of course change all “engine” calls to “get_engine()” everywhere.

Okay, let’s run again….

Wuhu! It works. We run our first integration test using the test containers library.

It was not so easy as described in the example but we did it.

The cool thing is that you now have full control of your MySQL instance container! You can stop it, reload it, kiss it …all this from your codebase.

You also don’t need a third-party script like “docker-compose”. Since testcontainers are a built-in solution, all you need now is to clone your repository, make sure you have docker installed on your machine, and run “pytest”.

And that’s a real point for testcontainers library.

Conclusion

Whether you choose docker-compose or testcontainers, those two solutions are great for integration testing. We will summary our thinking on those two solutions here :

Docker-compose :

Pros:

• Easy configuration
• Fast to run
• No change in the codebase: all is about the configuration file
• In the case of a database, you can easily create a volume to an init.sql script (In some cases it can be a preferred way over database migration, especially if you are not a great ORM supporter)

Cons :

• Static, once you launch it, you cannot really manipulate the containers within your code, unless using an external call like “os.system” or “subprocess” in python OR using a port to Unix socket as pydocker does)
• Your test suites are now dependent on a third-party script, and you need to run it each time before your tests

Testcontainers :

Pros :

• Built-in solution: once you configure it, you run it. Take a look at the GitHub action workflow of the testcontainers branch for example…and compare it to one of the docker-compose branches…we saved a step here! Because pytest will directly run the containers for us by calling the testcontainers API
• Control your containers from your tests!

Cons :

• Change your codebase (and that’s a hard con in my opinion): if your test suite only needs a running database instance…prefer the docker-compose solution
• Hard to make it works (in my opinion). If a test library implies changing code that is outside of the test scope…It might be because the code is not good (I am not saying that the code that we present is the ideal code far from that)…but to be honest it is the first time that I have to make such a change to get it to works.
• Take longer to run your test suite, since the test is now responsible to launch a container, it suffers from a bit of delay for the current version (3.4.1)