Monitoring AWS Batch Jobs using CloudWatch, Part 2: Testing SNS with LocalStack

In Part 1 of this series, I covered how I set up a notification system to alert my application of status changes to AWS Batch jobs. In this article, I’ll cover how I used LocalStack to emulate and test this behavior locally.

LocalStack is a wonderful open source project that emulates the behavior of a number of AWS services. This makes local development and testing much easier, as you can build with the confidence that when you deploy, all the pieces will work together.

In my case, I wanted to emulate as much of the following behavior locally as possible:

1. The web application submits a job to AWS Batch.
2. Each time a job changes status (from Submitted to Running to Succeeded/Failed), CloudWatch Events send a message to an SNS Topic.
3. The application, which is subscribed to the SNS Topic, receives the message and reacts accordingly.

Unfortunately, LocalStack does not currently provide support for AWS Batch, so we wrote a service to emulate its basic behaviors in our test environment. This article will not cover that behavior. Instead, I’ll focus on how we emulated the CloudWatch Event messages and SNS Topic subscription.

Getting LocalStack Running

Our application runs on Docker, so we decided to use the LocalStack Docker image to run the various services. To bring up the service, just add the following to your docker-compose.yml under services:

localstack:
    image: localstack/localstack
    environment:
      SERVICES: sns, s3
      DEFAULT_REGION: us-east-1
      HOSTNAME: localstack
      EXTERNAL_HOSTNAME: localstack
    ports:
      - "4567-4583:4567-4583"
      - "8080:8080"

For the use case we’re discussing here, we only need the SNS service, but I’ve also included S3 to demonstrate configuring and testing more than one AWS service. The mock web console is available at port 8080, and 4567–4583 represents the range of ports for each AWS service. The LocalStack documentation lists the port for each service here.

The built-in web console provided by LocalStack

When running docker-compose up, LocalStack will be brought up in addition to your other services. Note that the container should be brought up in the same network as your web service or any other service that needs to interact with it.

Now that we’ve got LocalStack mocking S3 and SNS, we can do some quick testing with AWS CLI. Interacting with LocalStack is almost identical to normal interactions with AWS services, you just need to specify the correct endpoint option:

aws --endpoint-url=http://localhost:4575 sns create-topic --name test-topic

If you’d like to use LocalStack for every interaction with AWS services in your local environment, you can specify the endpoint in your configuration files. In our Rails app, we added the following lines to our development.rb and test.rb files:

Aws.config[:sns] = {
  endpoint: 'http://localstack:4572',
}
Aws.config[:s3] = {
  endpoint: 'http://localstack:4575,
  force_path_style: true,
}

Topics, Subscriptions, and Messages, Oh My!

With the above configuration in place, any interaction with SNS and S3 will use the LocalStack mocks instead of AWS’s services, so we can set up our Topic and subscribe our endpoint that was created in Part 1.

client = Aws::SNS::Client.new
topic = client.create_topic(name: 'batch-job-status')
client.subscribe({
  topic_arn: topic.topic_arn,
  protocol: 'http',
  endpoint: 'http://localhost:8000/receive_sns',
})

Unlike the actual SNS service, LocalStack does not require a subscription confirmation, so our subscription is all set up and ready to receive messages published to the Topic.

Since CloudWatch Events will be responsible for publishing messages when a Batch job changes status, we want to emulate that structure locally. The full Batch Job State Change message structure can be seen here. For this demo, I’m mostly concerned with the status and jobId attributes, so I’ve slimmed down the message structure.

client.publish(
  topic_arn: topic.topic_arn,
  message: { 
    detail: { 
      status: 'FAILED', 
      jobId: '1234',
      jobName: 'test-batch-job',
    } 
  }.to_json
)

Since the web application is subscribed to the SNS Topic, the message appears in our server logs soon after it’s been published to the Topic. Hooray!

I, [2018-04-24T19:44:16.232866 #18]  INFO -- : Notification Message Received: {"detail":{"status":"FAILED","jobId":"1234","jobName":"test-batch-job"}}

Wrapping Up

Now that the various AWS services are hooked up, the receive_sns endpoint can be updated to process the message and update the job status accordingly. In the event a job succeeds, the application will submit the subsequent job to Batch. In the event of a failure, appropriate notifications can be sent or retry strategies employed. I can also update the service that emulates Batch so that it publishes the correct status message to the SNS Topic as each job is run so the integration tests ensure the messages are processed correctly and the application reacts accordingly.

Overall, we’ve been pretty happy with this implementation. There is very little setup work required, and we were able to avoid a lot of environment-specific code while still ensuring that our test and development environment behaved similarly to production.

If you’re interested in seeing LocalStack in action, I put together a quick demo app here. This is a Phoenix app (so the web code is different from the Rails code in this blog post), but it does demonstrate the docker-compose configuration and shows how to manage the various AWS services using the AWS CLI. Enjoy!