Create SQS Consumer by nodejs, typescript, localstack (using sqs locally), and terraform

Have you ever tried to create some POC with AWS services but you don’t want to actually pay for particular services at the moment?

LocalStack and Terraform might be the answer for you.

Localstack is a dockerize image that allows you to replicate most AWS Services locally such as SQS, SNS, SES, Lambda, and so on.

Terraform is an infrastructure as code tool. It allows you to config Cloud Services including Google Cloud Platform and Amazon Web Service by writing the .tf file.

This article will demonstrate how can we implement AWS Services locally with the Localstack and Terraform to do some experiments.

Assume that we have the requirements about creating a queue consumer (SQS) and do some business logic as the diagram below written in TypeScript and using NodeJS as runtime.

figure 1. queue service architecture design

From the architecture above, this system allow users to enqueue message to the SQS. Then messages would be consumed from SQS by SQS-Consumer and do some business logic later.

Dependencies

• docker-compose
• terraform
• nodejs (version 18)
• typescript

Project Setup

• Setup typescript Repo

In this case, we use stemmlerjs/simple-typescript-starter as initial setup for NodeJS and Typescript

git clone https://github.com/stemmlerjs/simple-typescript-starter queue-service

According to the above command, it will clone typescript project from the repo and rename it as queue-service

• install dependencies

cd queue-service
npm i

• install sqs-consumer ,aws-sdk and dotenv

npm i sqs-consumer
npm i aws-sdk
npm i dotenv

• to start node application in development mode

npm run start:dev

# stop the service by ctrl+c

• Install aws-local command

pip3 install awscli-local

This command is used to test localstack via cli

• Create docker-compose.yaml file under the project folder

version: '3.8'

services:
    localstack:
        container_name: "localstack_main"
        image: localstack/localstack:latest
        environment:
            - SERVICES=sqs,ses
            - LAMBDA_EXECUTOR=docker_reuse
            - DOCKER_HOST=unix:///var/run/docker.sock
            - DEFAULT_REGION=ap-southeast-1
            - DEBUG=1
            - DATA_DIR=/tmp/localstack/data
            - PORT_WEB_UI=8080
            - LAMBDA_DOCKER_NETWORK=host
        ports:
            - "53:53"
            - "53:53/udp"
            - "443:443"
            - "4566:4566"
            - "4571:4571"
            - "8080:8080"
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - localstack_data:/tmp/localstack/data
        network_mode: bridge

volumes:
    localstack_data:

The YAML file above defines services that are used in this article such as SQS. After you create this file, the project structure should look like this.

figure 2. docker-compose.yaml

• Start the localstack locally by using docker-compose

docker-compose up -d

• Create SQS Queue locally

SQS (queue) could be either created by awslocal/aws CLI or terraform

for example:

awslocal sqs create-queue --queue-name pilot-queue

figure 3. create awslocal sqs via cli

However, if you do the CLI as above, it would be a bit difficult for you to reproduce on other environments such as development, staging and production.

Since terraform is an infrastructure as code, it allows you to create a .tf file and run the service provisioning by simple commands.

• create main.tffile

provider "aws" {
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
  access_key                  = "mock"
  secret_key                  = "mock"
  region                      = "ap-southeast-1"
  endpoints {
    sqs = "http://localhost:4566"
  }
}

resource "aws_sqs_queue" "local_queue" {
  name = "local-queue"
}

• run provisioning script

$ terraform init
$ terraform validate
$ terraform plan
$ terraform apply 
// this article will not cover how to implement the terraform

figure 4. Provisioning localstack SQS by terraform

sqs queue url from the main.tf file would be

• http://localhost:4566/000000000000/local-queue

try to send message via CLI to validate sqs

awslocal sqs send-message --region ap-southeast-1 --queue-url http://localhost:4566/000000000000/local-queue --message-body '{"command":["do-something"]}'

firgure 5. Send message to localstack SQS via awslocal cli

note if you are using the actual aws-cli, replace awslocal with aws instead

Using terraform would be much easier if you want to provide service in other environments.

Create .env file

NODE_ENV=local
QUEUE_URL=http://localhost:4566/000000000000/local-queue
QUEUE_REGION=ap-southeast-1

Create SQS Consumer

• go to src folder and remove other files and keep only index.ts file

figure 6. remove unnecessary files

• Create SQSConsumerFactory Class in src/infra/sqs.consumer.factory.ts

figure 7. sqs.consumer.factory.ts

According the code above, the FactoryMethod is implemented just in case that there might be multiple queues that need to be created.

Create Config File

• create config variables in src/config.ts which loads from enviroment variable by using dotenv library.

figure 8. Config file

Implement Business Logic

• implement business logic in src/index.ts file

figure 9. index.ts

From the index.ts file, when the program start, it will called startConsume() function. Inside startConsume() function, sqsConsumer (pilotQueue) is created with queueUrl which we retrieve from the local-queue config and messageHandler function which will invoke after message is consumed from SQS. In the messageHandler() function, it will parse json only if message input is string of JSON. You guys can add business logic after the message parsing finished.

Final Project Structure

figure 10. Project Structure

Start the application and test

# terminal 1: start the app
npm run start:dev


# terminal 2: enqueue message to sqs by cli

awslocal sqs send-message --region ap-southeast-1 --queue-url http://localhost:4566/000000000000/local-queue --message-body '{"command":["do-something"]}'

awslocal sqs send-message --region ap-southeast-1 --queue-url http://localhost:4566/000000000000/local-queue --message-body 'only string'

End Result

firgure 11. End Results

According to the result above, you can implement business logic after the `message.Body` is consumed.

Source Code Repo

You guys can see the source code on this link

https://github.com/ccl-abacus/queue-service-node-js

At this point, I hope you understand the basic idea how to implement Localstack with Terraform locally. You can config/enable services on doceker-compose.yaml . Also you can manage or provisioning resources via terraform file as well.

Enjoy Coding!! Cheers!!

PS. this is the first article that I write in last two years. I apologize for any mistake that I made in this article. I hope this article would be benefit to you guys more or less.

BTW, SCB Abacus which is the company that I am working right now is hiring. If you guys are seeking for new challenges and you want to solve complicated problems, this is the right place for you.

Join Us: https://www.scbabacus.com/career

References

• stemmlerjs basic nodejs ts setup
• localstack github
• terraform + localstack guide