Testing with containers

In this article, I explain why, and how we use containers for testing purposes at Synthesio. For simplicity’s sake, most examples assume the project is written in Go, and use a MySQL database, and we will use Docker as the container application, but the method discussed can be and is used with any language or technology and more complex setups.

Dependencies are complex pieces of software, for a good reason. A database, for example, abstracts away the huge amount of complexity and features so our code doesn’t have to handle it by itself and can focus on business features while being simpler.

Testing code that use such a dependency, however, is a harder job than it seems made unnecessarily complex by most testing frameworks. One of the reasons for this is that when thinking, speaking, reading about testing, we could almost use the term unit-testing interchangeably. And all in all, unit-tests are badly adapted to testing code that directly use a complex dependency. Which isn’t surprising, given that the whole point of unit testing is to not use any dependency.

This is not the solution you’re looking for

In the current state of the art, the most commonly accepted solution for unit-testing a piece of code with a dependency is to fake it. To give the unit under test something that mimics the dependency and will behave as we want, so we can verify that the code is reacting as expected. This method, called mocking, is actually a bad solution to the problem at hand.

To be perfectly honest, faking the dependency is a good solution for a certain class of dependency. Unit-testing was born during the rise of object-oriented programming, and is unsurprisingly well-adapted to code that can take as dependency an interface of reasonable complexity, the mock being generally an in-memory, dumb version of the real dependency. A database for example, is generally too complex for this.

Let’s take a concrete example to illustrate the point. Golang has a library for the purpose of mocking a SQL database during tests, sqlmock. Here is an example of code, taken from the library's Github repository.

func TestShouldUpdateStats(t *testing.T) {
    db, mock, err := sqlmock.New()
    if err != nil {
        t.Fatalf("an error '%s' was not expected when opening a stub database connection", err)
    }
    defer db.Close()

    mock.ExpectBegin()
    mock.ExpectExec("UPDATE products").WillReturnResult(sqlmock.NewResult(1, 1))
    mock.ExpectExec("INSERT INTO product_viewers").WillReturnResult(sqlmock.NewResult(1, 1))
    mock.ExpectCommit()

    // now we execute our method
    if err = recordStats(db, 2, 3); err != nil {
        t.Errorf("error was not expected while updating stats: %s", err)
    }

    // we make sure that all expectations were met
    if err := mock.ExpectationsWereMet(); err != nil {
        t.Errorf("there were unfulfilled expectations: %s", err)
    }
}

If we think about it, the real interface between our code and the database isn’t the actual method called, it’s the query language itself. And it isn’t the query language that is mocked here, but simply the implementation code that is supposed to communicate with the database, leaving to the user the care to do the actual mocking, verification, etc.

The result of this is that the tests that use this kind of library generally do only half of the job, and generally not the part that is actually worth testing. Checking if a query wrapper really call the expected function is actually a near-useless job, akin to a test that verify the value of a constant.

A useful test would check if after calling the method, the value in the database is the expected one. Which would need faking the logic of the database itself and leave the implementation do whatever it wants with it. A useful test wouldn’t need to be completely re-written when the implementation of the method change even so slightly.

The problem is that a library that would implement such a feature would be hugely complex. It would probably have to implement complex logic to parse the SQL syntax, understand the queries, etc. If such a library existed, it would probably resemble more of a full-fledged in-memory SQL database than a mocking library. (I actually did exactly that using in-memory sqlite databases once upon a time.)

The amount of work needed would be huge to say the least. And even then, it wouldn’t be complete. It would have to understand vendor extensions, bugs, take into accounts versions of the target vendor database, etc, to be ultimately useful. Pouring such an amount of time in this is probably not worth it, when we have a simpler and ready solution: why not just use the real database?

This approach has existed for a long time too but it suffered multiple defaults, the most important being the need to install and configure a database for the tests themselves, which generally lead to a project needing a complex setup for something that should be simple and quick. Or you would need to maintain an active database in your organization for the purpose of running tests. And have one for each version used in production. Clumsy at best, not re-usable, and generally too many constraints.

Luckily for us, the situation changed in the last years, with the arrival of containers as an easy, simple and globally accessible solution. With a little not-even-complex tooling, using a real database in a container for testing is surprisingly easy, and lead to clean and concise tests.

Do or do not, there is no try

So, we want to run test that use a database with an actual database. How do we do that ?

First step is to have the database in a container. Let’s spawn a container for that. And throw in a docker-compose.yml for good measure.

version: "2.1"

services:
  mysql:
    image: mysql:5.7
    ports:
      - "3306:3306"

Now, running our tests is a simple matter of running docker-compose up before the tests, and using localhost:3306 as the database address. Dead simple. A little too naive, however.

For example, exporting ports that way is like opening a door to a world of conflicts, ad hoc conventions, etc, which in the long run would be hard to maintain. One solution is to run the tests from another container, linked to the mysql one so it can access the database using the container’s network.

For this, we will simply add an app container in our docker-compose.yml.

version "2.1"

services:
  app:
    image: golang:1.10
    links:
      - mysql
  mysql:
    image: mysql:5.7

Now, the tests must use mysql:3306 as the address, and the command for running the tests becomes docker-compose run --rm app /usr/bin/env bash -c "go test". Better on the operation side, but not something we want to type every time we want to run tests, configure CI, etc.

For simplicity’s sake, let’s put that in a Makefile. And while we’re at it, add a build command too so we can also build in the container.

exec = docker-compose run --rm app /usr/bin/env bash -c

.PHONY: build
build::
    ${exec} "go build"

.PHONY: test
test::
    ${exec} "go test"

Much better. Running the tests is back to a simple make test, the containers are spawned as needed without intervention, multiple projects can coexist without conflict, and the usage of Make or any other build too probably integrate with any complex workflow. All is well, our job here is done.

Until we actually run the tests….

# make test
[…]
--- FAIL: TestApp (0.00s)
 <autogenerated>:1: mysqltest: dial tcp 192.168.0.2:3306: getsockopt: connection refused
[…]

What is happening here? When it creates the containers, Docker is smart enough to wait until the mysql container is running before starting the app container, but most databases need a little warmup before being ready, so when the test code tries to connect to MySQL, the database is still initializing and cannot accept the connection.

One solution is to use a tool like https://github.com/jwilder/dockerize as the app container entrypoint. Dockerize will wait until the configured port is ready before running the container command. There is a little issue here: the golang container doesn't include dockerize.

Containers to the rescue! The simplest solution is to have a custom image that will do.

FROM golang:1.10.0

COPY entrypoint.sh /usr/local/bin/
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["/usr/bin/env", "bash"]

RUN curl -sSL "https://github.com/jwilder/dockerize/releases/download/v0.5.0/dockerize-linux-amd64-v0.5.0.tar.gz" | tar -xz -C /usr/local/bin

With entrypoint.sh being a file along these lines. (It can probably be replaced by something shorter in the docker-compose.yml's entrypoint option.)

#!/bin/bash
exec "$@"

Then, you can change the docker-compose.yml file to use your custom image (we will call it custom-golang) and dockerize.

version "2.1"

services:
  app:
    image: custom-golang:1.10
    links:
      - mysql
    entrypoint: dockerize -timeout 20s -wait tcp://mysql:3306 entrypoint.sh
  mysql:
    image: mysql:5.7

OK, we have a database container ready for action, it’s time to code! Let’s do something that resemble an actual test.

func TestFoo(t *testing.T) {
    // Prepare the connection to the database container.
    db, err := sql.Open("mysql", "root:@tcp(mysql:3306)")
    if err != nil {
        t.Fatal("connecting to database server:", err)
    }
    defer db.Close()

    // Create the database and tables necessary.
    _, err = db.Exec(`
        CREATE DATABASE app;
        USE app;
        CREATE TABLE foo ( id INTEGER UNSIGNED PRIMARY KEY );
    `)
    if err != nil {
        t.Fatal("initializing database:", err)
    }

    // Call the tested function.
    result := foo(db)

    // Check the result.
    var expected = "bar"
    if result != "bar" {
        t.Errorf("unexpected output: wanted %s, got %s", expected, result)
    }
}

While encouraging, this code has a number of problems that we need to address before being reliable. The first of them is that it will only work once, as the created base is persistent and the test will fail if the database already exists. Additionally, we won’t be able to run tests in parallel either.

We could add a DROP statement before the database creation, but it would only solve half of the problem. A better solution would be to generate a random name before actually running the query and use this as the database name.

name, _ := random.Alpha(10)

_, err = db.Exec(fmt.Sprintf("
    CREATE DATABASE `%[1]s`;
    USE `%[1]s`;
    CREATE TABLE foo ( id INTEGER UNSIGNED PRIMARY KEY );
", name))

Good enough. Although a little rough, this solution works and solve all the problems at hand. We can refine it further by moving the related code in a dedicated helper so the test itself is cleared of unnecessary code, and each test can use it independently.

func Spawn(t *testing.T, address, schema string) *sql.DB {
    // Prepare the connection to the database container.
    db, err := sql.Open("mysql", fmt.Sprintf("root:@tcp(%[1]s)", address))
    if err != nil {
        t.Fatal("connecting to database server:", err)
    }

    // Create the database and tables necessary.
    name, _ := random.Alpha(10)

    _, err = db.Exec(fmt.Sprintf(" CREATE DATABASE `%[1]s`; USE `%[1]s` ", name))
    if err != nil {
        t.Fatal("initializing database:", err)
    }

    // Load the schema.
    _, err = db.Exec(schema)
    if err != nil {
        t.Fatal("loading schema:", err)
    }

    // Return the database and a cleaning 
    return db
}

func TestFoo(t *testing.T) {
    // Create a firesh database for use in this test.
    db := Spawn(t, "mysql:3306", "CREATE TABLE foo ( id INTEGER UNSIGNED PRIMARY KEY )")
    defer db.Close()

    // Call the tested function.
    result := foo(db)

    // Check the result.
    var expected = "bar"
    if result != "bar" {
        t.Errorf("unexpected output: wanted %s, got %s", expected, result)
    }
}

Another big improvement would be to load the database creation queries directly from a file, so the schema and fixtures can be shared between different tests and won’t pollute the test code.

func Spawn(t *testing.T, address string, fixtures ...string) *sql.DB {
    // Prepare the connection to the database container.
    db, err := sql.Open("mysql", fmt.Sprintf("root:@tcp(%[1]s)", address))
    if err != nil {
        t.Fatal("connecting to database server:", err)
    }

    // Create the database and tables necessary.
    name, _ := random.Alpha(10)

    _, err = db.Exec(fmt.Sprintf(" CREATE DATABASE `%[1]s`; USE `%[1]s`; ", name))
    if err != nil {
        t.Fatal("initializing database:", err)
    }

    for _, fixture := range fixtures {
        Load(t, db, fixture)
    }

    // Return the database and a cleaning 
    return db
}

func Load(t *testing.T, db *sql.DB, fixture string) {
    raw, err := ioutil.ReadFile(fixture)
    if err != nil {
        t.Fatalf("reading fixture %s: %s", fixture, err.Error())
    }

    // Load the schema.
    _, err = db.Exec(string(raw))
    if err != nil {
        t.Fatalf("loading fixture %s: %s", fixture, err)
    }
}

func TestFoo(t *testing.T) {
    db := Spawn(t, "mysql:3306", "testdata/schema.sql")
    defer db.Close()

    // Call the tested function.
    result := foo(db)

    // Check the result.
    var expected = "bar"
    if result != "bar" {
        t.Errorf("unexpected output: wanted %s, got %s", expected, result)
    }
}

Conclusion

This testing method can be refined a bit more by adding a few tricks that won’t be covered here, like cleaning the database after a successful test, or adding templating to the fixtures. It can be adapted to almost any kind of database, or even other types of dependencies like message brokers, other services, etc.

Among the downsides is the fact that it’s actually slower than something like a mock. Spinning up the container is cheap but non-negligible and loading huge fixtures can be long, but all things considered it is often a small price to pay in comparison to the correctness and actual usefulness of the tests that implement this.

Feel free to reach out if you want more details, have questions, or just want to chat. I would love to hear your opinion on the subject.

Originally published at https://tech.synthesio.com/articles/2018-03-05-testing-with-containers/