How to test Kinesis with LocalStack

Background

Henrique Ribeiro Rezende and I were working on a story where we needed to listen to a Kinesis stream. Our service that listens to the Kinesis stream is written using Java and SpringBoot.

So what do we do now? We had to get this up and running. The AWS documentation already covers how to setup a listener, including code examples.

What was missing then?

Documentation on testing. We needed to have an end-to-end test that would:

1. Spin up the application locally.
2. Setup a Kinesis stream.
3. Put a message on the stream.
4. Assert that our listener has received the message and processed it.
5. Run irrespective of the environment.

All this needed to be done with minimum to none number of manual steps. We decided to use Localstack for this purpose since it is the easiest to setup and works really well with Docker.

Code

Kinesis Listener

The listener in our application has been implemented using SmartLifecycle from the Spring framework. This is because:

1. We want to start the listener in a separate thread.
2. SmartLifecycle gives us better control over the process and specify when the listener should start/stop as the context is refreshed. It also allows for a graceful shutdown.

Lets look at the start method of KinesisListener.java.

Credentials for test and production environments are different. Hence, we have to provide the credentials and the CloudWatch config explicitly (lines 10,12,18,19). If you were not writing any test for this particular functionality then all of this config would be picked up automatically because you would have configured it in the same AWS account where this application is deployed.

The Test

The complete test can be found here. Lets look at it in detail.

LocalStack provides test runners both with and without Docker. We went ahead with the Docker Runner as it just requires Docker running locally instead of having Localstack installed every time the tests are run. Also LocalStack takes care of startup and teardown of the Docker container itself.

Since we are using this test runner, we have to add two Rules (lines 7–10) that help us leverage the benefits of @RunWith(SpringRunner.class).

Following is the code for the test method.

We have to wait for the stream to be created before we proceed with putting a message on it. It takes a while since it will initialize the DynamoDB tables it needs (lines 6–8).

Similarly, there is some waiting needed for reading the message from the stream as well (line 23).

AWS Configurations

You will notice two files named AWSConfig.java. One under test and the other under src. This is because we need to provide separate set of credentials for the test and production code. In our system, we define a DynamoDB client so Kinesis can store its checkpoint information, CloudWatch client so Kinesis can report metrics and of course a Kinesis client to connect to the Kinesis stream. Based on the Profile annotation, the appropriate class is picked up.

In the file …/test/AWSConfig.java, we used DockerTestUtils to get the credentials and the DynamoDB/Kinesis clients provided by Localstack, whilst in …/src/AWSConfig.java we keep our production credentials.

Hope this saves some of your time when you write a similar test!