Testing in Golang(Part 2) — Database Integration Tests
In the second part of this blog, I would like to focus on how you can create integration tests in Go. When I started researching tests I found a library named go-sqlmock. I started to write some tests but I wasn’t happy with the confidence they give me. I wasn’t sure if queries would work or not as it’s only mock. I started to think about what I expect from them and defined a checklist that the setup should fulfill.
Requirements for an integration test setup:
- Tests should be fast — I don’t want to wait a few minutes to check if a small change is breaking tests or not.
- Tests should run on every save to make sure I get feedback as soon as something breaks.
- Always create the database from scratch — Run migrations and seed with data and finally destroy container after failed or a successful run.
- Run tests in parallel — each test suite should have its own Postgres image. I should be able to run the same test suite more than once and don’t be afraid that tests fail.
- Run integration tests run in continuous integration pipelines ie. GitHub Actions
Then I started to look for tests with the docker PostgreSQL image. My first thought was to use docker-compose. I did more research and found a project named dockertest created by Ory. The library allows you to fetch images of Postgres, MySql, Kafka, MongoDB, or any image you would like to test. As it’s configured in code I can run it incredibly fast without any effort.
My expectations were fulfilled and I love to use that library. If that sounds exciting to you let’s start with the example.
Integration Tests Example
As it’s always better to have the full picture I’d like to go ahead and explain the full flow.
You can follow the example in the repo if you get lost at any point:
https://github.com/suuum/golang-integration-tests-example
1. Create and configure a container image.
The function you see pulls and creates containers with a defined setup. It’s possible to configure containers and check all properties, which is handy while debugging. As you can see, the function has a parameter exposed port set in PortBindings. This allows the program to assign a container with a random port and run multiple instances in parallel without worrying about the currently used port.
2. Make the connection to DB
To establish a DB connection you can use hard-coded values as all DB credentials are the default for Postgres. You only need to provide the exported port as it’s random in each run.
As it can sometimes happen that Postgres isn’t ready, I recommend implementing a retry mechanism and being resilient. Otherwise, from time to time, you will get a connection error and your test will fail during setup.
3. Create a simple model
To create a DB entity you need to define a model with “gorm” attributes. This code allows you to specify more properties like column names, keys, etc. The following example name was changed to use Postgres naming conventions.
To define a table name for the entity you need to implement the TableName function with the needed value.
4. Create migrations
Gorm provides the feature to migrate all specified entities in the Automigrate method. This is ok for article purposes, but in the production system you need more control over migrations and to define it with multiple files that contain migrations to build and destroy the database.
4. Seed data
To populate entities you must implement functions that are easy to extend. The code below defines a seed struct with a name and func to execute Create method.
Adding new data is possible by extending the array in func All(). As entity types are not defined, you can add any data type you want.
5. Seed data
The function iterates through the specified data in function All().
6. Tests suite
The suite is usually implemented in three parts.
Before Suite
Function to init all necessary parts:
- get a random port (in the production system it’s better to use ports from the range 1024–65535)
- pull the image and wait till it’s available
- open Postgres connection
- run migrations with specified schema
- seed database with test data
- init repository struct
Describe
Place where test cases should be added.
AfterSuite
Destroys the container with the database or any other image.
Executing the tests:
Start by simply checking if the example is working. Running a simple test:
It’s done in 2 seconds, so it’s a pretty good result.
Let's try running 5 test cases at the same time:
This takes longer but you still get feedback within a sufficient timeframe.
Tip for Integration Testing:
Be careful with seeding data — it’s better not to depend on the same entities in every test suite. As soon as you start changing code in one place you would need to adjust every other test that depends on that data.
A setup like this works best when the system doesn’t have many components that interact together. For creating clusters or many interacting components I would use docker-compose for local testing and do a similar setup in the CI pipeline.
GitHub Action
Simple Github action that runs on every push to your repository:
As you can see GitHub Action doesn’t need to set up any services since everything is handled by the dockertests library. The CI code is clean and easy to understand. Postgres is only one example and you can use other docker images like MongoDB, Kafka, Redis, etc. Using custom docker images is also possible.
If you want to check the complete example, simply visit the repository:
https://github.com/suuum/golang-integration-tests-example
Sources:
https://github.com/ory/dockertest
https://github.com/go-gorm/gorm
