Functional Testing DynamoDB Services with a CI/CD Pipeline and Containers

Dealing with state in “the cloud” is often more trouble than it is worth. When I talk about architecture and infrastructure with my clients they only ever have one real rule, “We only want to use managed databases.” I get it. DynamoDB is one of my favorite database services. I’ve used it in projects taking a few requests per second and others taking tens of thousands.

My favorite thing about DynamoDB is that it provides declarative performance. Users specify and pay for throughput directly. In contrast to other managed or self-hosted databases where performance is an emerging property. That declarative interface lets you plan and specifically pay for throughput.

AWS accounts and DynamoDB itself are centralized and stateful (duh). This is a problem when developers are working from their laptops and multiple codebases / tests are accessing data stored in the same set of tables. It is a problem because different versions of the software might be making inconsistent changes to the schema, or storing incompatible data, or changing records that are required by other versions or functional tests. When that happens the developer impact is significant, tests fail (or pass) when they shouldn’t, and the whole system feels fragile.

I’m picking on DynamoDB a bit here. These are problems for every database technology in centralized development and test environments. However, DynamoDB could not run locally until recently and so very few adopters do so. Instead they build separate AWS accounts, and pay for separate tables and throughput. Those tables are arbitrarily provisioned and a busy afternoon might result in functional tests that brownout provisioned throughput and cause random test failures. This works for lots of people, but there is a better way.

Sometime ago AWS released a local version of DynamoDB with a SQLite based implementation. This is a welcome improvement and including it with your development environment is a must. Using it for functional testing takes just a little bit more effort.

Addressing the Gap

Imagine a build-test-deploy pipeline that can run a full suite of functional tests — including state mutations — without worrying about cleaning up those state changes. Imagine being able to functional test against databases in different states.

We’ve got those tools for other stacks, but I wanted to see something container-native for DynamoDB. I spent an afternoon putting together dynamodb-snapshot-containers.

This project help you create local DynamoDB snapshots and save them as Docker images. A user customizes the snapshots by writing a little script with AWS CLI commands. The tool runs those scripts in a container and saves the resulting customized DynamoDB container as a new image.

When you create a container from the snapshot image the local DynamoDB instance will start with the state declared in the script. This isn’t particularly advanced tech, but it can save you time creating Docker images for use in particular phases of your CI/CD pipeline.

Consider the following DroneCI pipeline:

pipeline:
build:
image: golang
commands:
- go get
- go build
- DB=dynamodb-step1 go test functional1
- DB=dynamodb-step2 go test functional2
services:
dynamodb-step1:
image: myprog/dynamo-snap:user-bob-created
dynamodb-step2:
image: myprog/dynamo-snap:user-bob-admin

A user can create the snapshots required for the tests with scripts like the following example

#!/bin/sh
# Add a known table
aws dynamodb create-table \
--endpoint-url http://localhost:8000 \
--region us-west-2 \
--table-name acl \
--attribute-definitions \
AttributeName=customer,AttributeType=S \
AttributeName=resource,AttributeType=S \
--key-schema \
AttributeName=customer,KeyType=HASH \
AttributeName=resource,KeyType=RANGE \
--provisioned-throughput \
ReadCapacityUnits=5,WriteCapacityUnits=5

# Add some known state
aws dynamodb put-item \
--endpoint-url http://localhost:8000 \
--region us-west-2 \
--table-name acl \
--item '{"customer": {"S": "Bob"}, "resource": {"S": "ACCOUNT-1234"}, "access": {"S": "Admin"}}'
aws dynamodb put-item \
--endpoint-url http://localhost:8000 \
--region us-west-2 \
--table-name acl \
--item '{"customer": {"S": "Bob"}, "resource": {"S": "ACCOUNT-4321"}, "access": {"S": "User"}}'
aws dynamodb put-item \
--endpoint-url http://localhost:8000 \
--region us-west-2 \
--table-name acl \
--item '{"customer": {"S": "Ray"}, "resource": {"S": "ACCOUNT-4321"}, "access": {"S": "Admin"}}'

And then build the snapshot with the build-snap.sh script:

./build-snap.sh \
example-script.sh \
awesomeproject/db-test-suite:acl-case2 \
"jeff@allingeek.com" \
"ACL project testing: 2 admin, 1 user"

The project requires a POSIX shell and the docker command line client. The build and resulting snapshot images require a Docker daemon capable of running Linux containers.

If you find the project helpful I’d love to hear from you. Or if you have enhancements that you’d like to share I’m always interested in pull requests.