Photo by Émile Perron on Unsplash

A Node.js introduction to Amazon Simple Queue Service (SQS)

Derek Woods
Apr 7 · 6 min read

Amazon SQS is a message-queuing system that allows you to write distributed applications by exposing a message pipeline that can be processed in the background by workers. If you are in the need of a tool to facilitate distributed applications, event-driven architecture or event processing then SQS may be a solid choice to add to your tech arsenal. Compared to other popular message queues, SQS does not have up-front costs and instead you pay for what you use. This makes experimenting with SQS a compelling value-add service for your application.

There are 3 main components to consider when working with message queues:

  1. The messages that get put onto the queue
  2. The specific queue the messages go into
  3. One or more workers that process messages on the queue

The following diagram demonstrates the flow of messages:

SQS Message flow

The application produces messages during its typical request cycle ,and once the messages are added onto the queue, the request terminates. At this point there is no confirmation that the message has actually been handled, only an effective promise that the message will be eventually processed by a worker.

In the meantime there are one or more workers constantly monitoring the queue waiting for new messages to process. Once a new message is found a worker will perform the work to handle the message, and once completed the message itself will be removed from the queue. Typically a message produces artifacts in the form of data updates, data inserts or requests to other systems but the end result is the message has been handled and removed from the queue.

Pre-requisites

You will need an AWS account and a valid AWS credentials file. You’ll need to know your Account ID which you can find at the top of https://console.aws.amazon.com/billing/home?#/account

You can checkout the code from Github at https://github.com/methodin/SQS-node-intro


Setting up a queue

To setup a queue you first login to AWS and navigate to https://console.aws.amazon.com/sqs/home?region=us-east-1

Click the Get Started Now button on the page. On the subsequent form you will name your queue, leave the type as Standard, then click Quick Create Queue. Once the queue is created you will be taken to the queue list screen where you will see the URL property — copy the URL property for use in Node.js code:

The URL property is what you will use in your Node.js code

Adding a message

Let’s assume we had a billing system that should charge a credit card when a user places an order. In traditional systems you would either process the billing request at the time of order submission or update a database table to signify an order is ready to be processed by a cron job that would do the actual billing action.

Using SQS we can instead add a new billing message onto the queue with enough data to allow the worker to perform its task.

{
"order_id": 1234,
"date": "2019-03-12T04:55:20.793Z"
}

The line between including all relevant data in the message itself versus looking data up in the worker is not a finite one. You must balance risk, cost and execution time when determining how much data to include or exclude.

Now that we have a message structure we need to send it to SQS. To do this we must import the AWS SDK library, create an SQS client then send the message.

// Load the AWS SDK for Node.js
const AWS = require('aws-sdk');
// Set the region we will be using
AWS.config.update({region: 'us-east-1'});
// Create SQS service client
const sqs = new AWS.SQS({apiVersion: '2012-11-05'});
// Replace with your account id and the queue name you setup
const accountId = '1234567';
const queueName = 'test';
// Setup the sendMessage parameter object
const params = {
MessageBody: JSON.stringify({
order_id: 1234,
date: (new Date()).toISOString()
}),
QueueUrl: `https://sqs.us-east-1.amazonaws.com/${accountId}/${queueName}`
};
sqs.sendMessage(params, (err, data) => {
if (err) {
console.log("Error", err);
} else {
console.log("Successfully added message", data.MessageId);
}
});

If you are sending more than one message at a time you will want to use sendMessageBatch which will reduce the cost and delivery time.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SQS.html#sendMessageBatch-property

Running the script above should yield a new message in your SQS. If you select the queue, right click and hit View/Delete Messages you should see a new message in the queue.

A new message is waiting in the queue

Creating a worker

With our message now sitting in the SQS queue we must now create a worker than can process the message. Similar to the sendMessage code above we have to import the AWS SDK library and create an SQS client. Once the client is setup we call receiveMessage to find a message to process:

// Load the AWS SDK for Node.js
const AWS = require('aws-sdk');
// Set the region
AWS.config.update({region: 'us-east-1'});
// Create SQS service object
const sqs = new AWS.SQS({apiVersion: '2012-11-05'});
// Replace with your accountid and the queue name you setup
const accountId = '131275825514';
const queueName = 'test';
const queueUrl = `https://sqs.us-east-1.amazonaws.com/${accountId}/${queueName}`;
// Setup the receiveMessage parameters
const params = {
QueueUrl: queueUrl,
MaxNumberOfMessages: 1,
VisibilityTimeout: 0,
WaitTimeSeconds: 0
};
sqs.receiveMessage(params, (err, data) => {
if (err) {
console.log(err, err.stack);
} else {
if (!data.Message) {
console.log('Nothing to process');
return;
}
const orderData = JSON.parse(data.Messages[0].Body);
console.log('Order received', orderData);
// orderData is now an object that contains order_id and date properties
// Lookup order data from data storage
// Execute billing for order
// Update data storage
// Now we must delete the message so we don't handle it again
const deleteParams = {
QueueUrl: queueUrl,
ReceiptHandle: data.Messages[0].ReceiptHandle
};
sqs.deleteMessage(deleteParams, (err, data) => {
if (err) {
console.log(err, err.stack);
} else {
console.log('Successfully deleted message from queue');
}
});
} });

If you ran the worker you would receive the message added in the first code segment, and now that you have the message, you are able to execute your billing strategy using the data contained in the message.

You can have as many workers running as you want. Ideally the number of workers would scale with the number of messages sitting in the queue ready to be processed. You can run workers in Docker containers using ECS, a traditional EC2 instance or even tie the SQS queue directly to a Lambda for processing. Depending on how you deploy your workers you can leverage auto-scale properties to make sure your workers scale with the number of messages in the queue. For purposes here we will just run the worker manually.


Next steps

As you can see setting up and integrating SQS into your application is pretty simple and opens up a lot of opportunities to add scaling and stability to your application — all without needing to worry about server setup.

There are plenty of other advanced use-cases and features that SQS provides, among them:

  1. Dead-letter queues for items that cannot be processed
  2. FIFO ordered queues for guaranteed ordering
  3. SQS Lambda streams to process SQS queue items automatically without running workers manually

All of these combined make SQS a compelling offering and one you should consider in your current, or a future, application.

Derek Woods

Written by

A software architect and general enthusiast thriving in the tech industry for over 15 years - I'm always looking at the cutting edge of technology.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade