Golang Integration Testing Made Easy

How we at Gojek implement integration testing in our development processes.

At Gojek, integration tests play a big role in our software development practices. This ensures that multiple parts of our application work together as expected and helps us meticulously solve problems.

In this blog, I’ll share nitty-gritties about the integration testing approach we implement in our daily software development process — especially backend development.

While this approach could be implemented in different programming languages, the flow helps us solve issues such as:

• Absence of mocking library for that particular dependency. In my case, it was Apache Kafka or when I wanted to test Redis and there was no mocking library for the development language I used.
• Even when there is a mocking library, situations where we want to make sure the query is sent correctly to the DB or when we use caching and want to ensure the cache is created/deleted.

Our Approach

The diagram above describes the flow of our integration testing. We utilise Makefile and docker-compose as our tools. Makefile simplifies testing command execution and Docker compose helps us run multiple containers that we use in our application. The flow would look like this:

Step-1: Create Makefile commands. This sets up our basic testing commands.

Step-2: Create or edit docker-compose.yaml. This defines application dependencies (DB, cache, message broker, etc). You can edit your docker-compose.yaml if you have a new dependency.

Step-3: Create an integration testing code in your favourite language.

Step-4: Execute the test command.

If you have a new use case to test, just repeat Step 3 and 4.

Let’s jump into an example.

Say we have a blog backend API server that will save an article, get articles by id, and get all articles. We will have an end to end test for the whole application flow and also the database interaction test. This is how our folder structure would look like:

[Note: Some folders and files are omitted]

├── article
│ ├── handler
│ ├── repository
│ │ └── article_postgres.go
│ ├── usecase
├── it
│ ├── article_repository_test.go
│ ├── e2e_test.go
│ ├── it.go
│ └── postgres_suite_test.go
├── server
│ ├── server.go

The code inside the server package will have a dependency on the handler, use case, and repository. The end to end testing will spawn the server, and we’ll send some requests and assert the response. The database interaction test will assert the method inside the repository package. You can find the full source code below:

arielizuardi/golang-backend-blog

The server code:

The database interaction code inside the repository package:

Create Makefile commands

Originally, Makefile is not only used to list out a set of directions to compile C or C++files, but also for other purposes. In our case, we list out a set of directives for our integration testing commands.

If you’re not too familiar with Makefile, check out this and this to get a better understanding.

Open your command-line interface

# Change directory to the root of your project directory
# Create a new file
$ touch Makefile
# Open your favorite editor to edit Makefile
$ nano Makefile

The Makefile will look like this

# we will put our integration testing in this path
INTEGRATION_TEST_PATH?=./it

# set of env variables that you need for testing
ENV_LOCAL_TEST=\
  POSTGRES_PASSWORD=mysecretpassword \
  POSTGRES_DB=myawesomeproject \
  POSTGRES_HOST=postgres \
  POSTGRES_USER=postgres

# this command will start a docker components that we set in docker-compose.yml
docker.start.components:
  docker-compose up -d --remove-orphans postgres;

# shutting down docker components
docker.stop:
  docker-compose down;

# this command will trigger integration test
# INTEGRATION_TEST_SUITE_PATH is used for run specific test in Golang, if it's not specified
# it will run all tests under ./it directory
test.integration:
  $(ENV_LOCAL_TEST) \
  go test -tags=integration $(INTEGRATION_TEST_PATH) -count=1 -run=$(INTEGRATION_TEST_SUITE_PATH)

# this command will trigger integration test with verbose mode
test.integration.debug:
  $(ENV_LOCAL_TEST) \
  go test -tags=integration $(INTEGRATION_TEST_PATH) -count=1 -v -run=$(INTEGRATION_TEST_SUITE_PATH)

Create docker-compose.yml

Docker compose will help us run multiple Docker containers easily. We will configure our docker dependencies in the YAML file. If you have not installed docker-compose, you can install it from this link.

version: "3.1"

services:
  postgres:
    image: postgres
    environment:
      POSTGRES_PASSWORD: mysecretpassword
      POSTGRES_DB: myawesomeproject
      POSTGRES_HOST: postgres
      POSTGRES_USER: postgres
    ports:
      - 5432:5432

This configuration tells docker-compose to run Postgres image. You can add other images depending on your use case.

Create integration test code

Now, we need to create integration testing for the above code. Let’s create e2e_test.go and article_repository_test.go file under it folder.

Execute test integration command from your CLI

// Run docker compose
$ make docker.start.components
// Run integration testing
$ make test.integration

You will see something like this:

DB_HOST=localhost DB_PORT=5432 DB_NAME=blog DB_USER=postgres DB_PASS=mysecretpassword DB_MAX_IDLE_CONN=10 DB_MAX_OPEN_CONN=200 DB_CONN_MAX_LIFETIME=30m ENV=local PORT=8080 \
  go test -tags=integration ./it -v -count=1
=== RUN   TestE2ETestSuite

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.1.10
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:8080
=== RUN   TestE2ETestSuite/Test_EndToEnd_CreateArticle
=== RUN   TestE2ETestSuite/Test_EndToEnd_GetAllArticle
=== RUN   TestE2ETestSuite/Test_EndToEnd_GetArticleByID
--- PASS: TestE2ETestSuite (0.18s)
    --- PASS: TestE2ETestSuite/Test_EndToEnd_CreateArticle (0.03s)
    --- PASS: TestE2ETestSuite/Test_EndToEnd_GetAllArticle (0.05s)
    --- PASS: TestE2ETestSuite/Test_EndToEnd_GetArticleByID (0.05s)
=== RUN   TestPostgresRepositoryTestSuite
time="2020-08-17T15:44:42+07:00" level=error msg="http: Server closed"
time="2020-08-17T15:44:42+07:00" level=info msg="shutting down the server"
=== RUN   TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_CreateArticle
=== RUN   TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_GetAllArticle
=== RUN   TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_GetArticleByID
--- PASS: TestPostgresRepositoryTestSuite (0.23s)
    --- PASS: TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_CreateArticle (0.07s)
    --- PASS: TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_GetAllArticle (0.07s)
    --- PASS: TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_GetArticleByID (0.08s)
PASS
ok   github.com/arielizuardi/golang-backend-blog/it 1.810s

Voila! We just finish our integration testing. 🖖

Want to read more stories from the vault? Check out our blogs!