SNS — SQS, and the MuleSoft liaison

When it comes to decoupling systems, varied messaging patterns come to our mind — each with their own uniqueness which encourages us to break down silos and achieve more together!

With the above thought in mind let’s take a moment to briefly look at two of the industry leading components under the AWS suite of messaging systems.

AWS Simple Notification Service (SNS)

• This managed AWS resource is central to the idea of a pub/sub model, wherein a publisher can publish messages to topics with subscribers being subscribed to those topics.
• In the event of a message being received on such a topic, all the subscribers of that topic will get notified.
• These subscribers can be as varied and as vast as your imagination might extend, starting right from Amazon Kinesis Data Firehose delivery streams to Lambda functions, Amazon SQS queues to HTTP/S endpoints, and AWS Event Fork Pipelines.
• Published messages are stored across multiple, geographically separated servers and data centers, so your data is safe and durable.

AWS Simple Queuing Service (SQS)

• This secure, durable, and available hosted queue lets us integrate and decouple distributed software systems and components.
• SQS allows greater control on what can be done with a message, including robust capabilities like visibility timeouts, delay queues and dead letter queues.
• SQS uses redundant infrastructure to provide highly-concurrent access to messages and high availability for producing and consuming messages.
• Choose to transmit sensitive data by protecting the contents of messages in queues by using default Amazon SQS managed server-side encryption (SSE), or by using custom SSE keys managed in AWS Key Management Service (AWS KMS)

Context and Ideation

In today’s blog we will introduce MuleSoft as a nexus between SQS and SNS services to achieve end-to-end integration capability utilizing the true prowess of asynchronous processing.

• In this blog, I will talk about a pub/sub approach to achieving decoupling and asynchronicity, wherein I have a single main SQS queue (main-stock-queue) and its DLQ (main-stock-dlq) subscribed to three SNS topics viz., GOOGL, GM and CVS.
• This setup ensures that whenever any message is published to any of the topics, the SQS queue will be notified instantly and can then process the message as desired, independently.
• This also sets the context for hassle free inter-app communication, with the backbone messaging capabilities managed entirely by AWS.

Underlying steps and strategy

1. We will start off with three SNS topics viz., GOOGL, GM and CVS. (We will also use a PG topic as a test topic for our SNS connector.)
2. We will be publishing a custom message to our desired SNS topic, as chosen during the publish.
3. A main SQS queue will be listening to all the three SNS topics and as soon as a message is received, will log a confirmation of the receipt.

Behind the scenes setup

The SNS topics

The main SQS queue and its DLQ

To achieve the said integration,

1. We need to subscribe the SNS topics to the main SQS queue, and
2. Setup something called an IAM policy (permission) on the main SQS queue that will allow it to receive messages from the subscribed SNS topics.

I have subscribed the three SNS topics to the main SQS queue and used the following IAM on that queue, to make the communication between SNS and SQS possible.

Associated IAM Access policy

Topic Subscriptions on the main SQS queue

General configurations and allied properties

Note - I am using Basic Connection type for both my SQS and SNS connectors, which requires you to generate AWS Access key (access key ID and secret access key)

Generating AWS Access Key

1. Click on the drop-down beside your AWS admin account name, from the top-right corner of the AWS Management console and select Security credentials
2. From the Your Security Credentials page, click on the drop-down beside Access keys (access key ID and secret access key).
3. Now just click on the button that says Create New Access Key, to generate a new AWS Access Key (keep the secret key safe, AWS allows you to download / save it only once, while it’s created!) to be used in Anypoint Studio.

I have used secure properties module to encrypt the AWS access key values and the SQS and SNS configuration parameters as follows,

config.yaml

Parameters Involved (encrypted using security module)

• aws.access-key → AWS access key ID
• aws.secret-key → AWS secret key
• sns.test-topic → ARN of the SNS test topic (arn:aws:sns:us-east-1:xxxxxxxx:PG)
• sns.arn-pattern → ARN pattern of the SNS topics (arn:aws:sns:us-east-1:xxxxxxxx:) to publish messages to
• sqs.arn → ARN of the main SQS queue (arn:aws:sqs:us-east-1:xxxxxxxx:main-stock-queue)
• sqs.url → URL of the main SQS queue (https://sqs.us-east-1.amazonaws.com/xxxxxxxx/main-stock-queue)

Setting up the SQS connector

Setting up the SNS connector

The Proof is in the Pudding!

Mule Flows involved

sns-publish

• In this flow, we will use the HTTP Listener component to receive a payload containing a custom “message” and the SNS topic name (“topic”) to publish this message to. Below is the payload being used

{
“message”: “This message is intended for <<topicName>>”,
“topic”: “<<topicName>>”
}

• A set variable component captures the topic name (“topic”)from the sent payload and a Transform Message is used to prepare (write) the payload as json message to be written to an SNS topic
• Finally, the SNS Publish operation is being used to publish the message to the mentioned topic (Topic arn), as shown below

sqs-message-trigger

• In this flow, we use the SQS Receive messages operation to continuously listen for newer messages on the main SQS queue (main-stock-queue). We need to set the Queue url to achieve this, as shown below

• Two Transform Message are used to extract the value of the customized message sent to the SNS topic, as follows

Processing verbose queue payload

Retrieving the actual message sent from an SNS topic

Putting it all together

I am sending the below customized message for topic CVS, via POSTMAN

This message makes its way to the SNS topic CVS, and immediately gets delivered to the SQS queue, (as the SQS is subscribed to this topic, as explained earlier). The verbose payload is received by the SQS, as shown below

The SQS Receive messages operation then immediately picks up this message from the SQS queue and the Message is extracted from the above payload in json form by our mule flow, and gets logged as shown below

The same can be tried for the two other SNS topics (GOOGL and GM) as well and inter-app error-free, highly scalable communication can be easily achieved.

That’s all for this blog, folks! Hope you find this article useful.