Using Localstack SNS and SQS for local testing

When we develop/update any application feature, it’s good to do a devbox testing. Devbox means developer machine. Quality Analysts and business Analyst validates if new changes are working fine in local. If everything works in local machine, only then push the changes to staging through CI/CD pipelines. It helps us to avoid lot of issues on staging environment. Also we can get early feedbacks from QA and BA.

When we use services like AWS for our application, we can use a mock service for testing it in dev machines . One such application that can help us is Localstack.

Localstack is a open source mocking framework for AWS cloud applications. You can run localstack docker container for integration testing and devbox testing. Also you can run this container in CI/CD pipeline for running integration tests.

Let’s take a simple example. Let’s say we have two microservices, order service and delivery service. order service sends order creation event to SNS. And delivery service subscribes to that topic via SQS queue. This can be tested in devbox using localstack. We don’t have to wait till the environment/services are setup in staging/test environments.

localstack

Let’s check how to setup localstack.

Setting up docker compose file

Add localstack in docker compose.yml file. LocalStack can be used to test various AWS services like S3, SNS, SQS etc. In the below example we will be using AWS SNS and AWS SQS services. AWS SNS is a messaging service. And SQS is message queuing service.

version: '3.0'

services:
  localstack:
    image: localstack/localstack:latest
    environment:
      - SERVICES=sqs,sns
      - AWS_DEFAULT_REGION=eu-central-1
      - EDGE_PORT=4566
    ports:
      - '4566-4597:4566-4597'
    volumes:
      - "${TMPDIR:-/tmp/localstack}:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"

Run docker-compose up -d for running the container in background.

You can check if the services are running by using docker ps or curl http://localhost:4566

For integration tests, you can use AWS SDK . Also you can use script with AWS CLI commands which you can run before start of the integration tests.

In the below example lets setup AWS service manually using AWS CLI commands

Setting up AWS CLI for testing

Setup AWS CLI if you do not have it installed. You can verify by using following command.

aws --version

Add dummy AWS creds for testing. It’s required for running AWS CLI commands locally.

aws configure set aws_access_key_id "dummy" --profile test-profile
aws configure set aws_secret_access_key "dummy" --profile test-profile
aws configure set region "eu-central-1" --profile test-profile
aws configure set output "table" --profile test-profile

You can also combine these commands into one command by using makefile as well.

Setting up SNS topic and SQS queue

Once the service starts running, you can setup the topic and queues which are required for testing. Below are some of the AWS cli commands that can be used.

1. Create SNS topic by using following command.

aws --endpoint-url=http://localhost:4566 sns create-topic --name order-creation-events --region eu-central-1 --profile test-profile --output table | cat

SNS topic creation

2. Let’s create SQS queue. This Queue can become a subscriber for AWS SNS topic.

aws --endpoint-url=http://localhost:4566 sqs create-queue --queue-name dummy-queue --profile test-profile --region eu-central-1 --output table | cat

SQS queue creation

3. To view the messages that is sent to the SNS topic, lets subscribe SQS queue to the topic.

Note: There are different ways to subscribe to the topic. Some endpoint types are email, sms, SQS, Lambda etc. You can refer to AWS document for more details. In the below example we used SQS for subscribing to the SNS topic.

aws --endpoint-url=http://localhost:4566 sns subscribe --topic-arn   arn:aws:sns:eu-central-1:000000000000:order-creation-events --profile test-profile  --protocol sqs --notification-endpoint http://localstack:4566/000000000000/dummy-queue --output table | cat

SNS subscribe command

4. You can receive events in SQS queue by using following AWS CLI command.

aws --endpoint-url=http://localhost:4566 sqs receive-message --queue-url http://localhost:4566/000000000000/dummy-queue --profile test-profile --region eu-central-1 --output json | cat

5. You can delete message from SQS queue.

aws sqs delete-message --endpoint-url=http://localhost:4566 --queue-url http://localhost:4566/000000000000/dummy-queue --profile localstack --region eu-central-1  --receipt-handle <message-handle>

6. Command to send events to SNS topic

aws sns publish --endpoint-url=http://localhost:4566 --topic-arn arn:aws:sns:eu-central-1:000000000000:order-creation-events --message "Hello World" --profile test-profile --region eu-central-1 --output json | cat

Send events to topic

7. Command to send events to SQS queue.

aws --endpoint-url=http://localhost:4566 sqs send-message  --queue-url http://localhost:4566/000000000000/dummy-queue --profile test-profile --region eu-central-1  --message-body '{
          "event_id": "7456c8ee-949d-4100-a0c6-6ae8e581ae15",
          "event_time": "2019-11-26T16:00:47Z",
          "data": {
            "test": 83411
        }
      }' | cat

Send events to SNS queue

Summary:

LocalStack is an open-source mock service for AWS services. It’s really easy to set it up with docker. You can integrate it with your application running locally for doing devbox testing and integration testing. And also can be used in CI/CD pipelines.