Publish/Subscribe Fan-Out Pattern in Serverless Architectures Using SNS, SQS and Lambda

In this article, we are going to learn Publish/Subscribe Fan-Out Pattern in Serverless Architectures Using SNS, SQS and Lambda.

Publish/Subscribe Fan-Out Pattern in Serverless Architectures Using SNS, SQS and Lambda

As you know that we can apply publish/subscribe and fan-out patterns with using different AWS Serverless services. By the end of the article, we will develop Hands-on Lab : Fan-Out Serverless Architectures Using SNS, SQS and Lambda.

I have just published a new course — AWS Lambda & Serverless — Developer Guide with Hands-on Labs.

Introduction — Fan-Out Serverless Architectures Using SNS, SQS and Lambda

Lets directly look at E2E architecture for our hands-on lab that uses to publish/subscribe fan out pattern. Basically we are following e-commerce architecture and use case is order processing.

Publish/Subscribe Fan-Out Pattern in Serverless Architectures Using SNS, SQS and Lambda

First of all, a customer using a web or mobile application and place an order. The client app sends this request to Amazon API Gateway endpoint. this endpoint is the “front door” of the application.

After that, API Gateway sends to order request to the first microservices which is Order Acknowledgment Microservice.

This microservices is use AWS Lambda function for computing.

This Order Acknowledgment Microservice does 4 thing;
1- first it verifies the request and generates the order Id
2- insert item order data into dynamodb table
3- Lambda sends a confirmation message back to api gateway with order Id

And lastly, this microservices publish a message to Amazon SNS which will be send downstream to different microservices. This message will be pub/sub messaging using SNS and this Order Acknowledgment Microservice will be the publisher of message.

In downstream microservices,

• we have notification microservices to send email and sms notifications to customer.
• we have inventory microservices to deduct products from warehourse.
• we have shipment microservices to ship products to customer.
• We can also add Data lake ingestion microservices to ingest all data generated from any process into their data lake for arbitrary analytics.

Of course we will use SNS and SQS in order to apply pub/sub fanout design pattern into our architecture. We will put individual SQS queues in front of our downstream microservices and use SNS to fan out messages to these queues.

Once the order data reach the queues from SNS, AWS Lambda downstream microservices automatically polls the queues extract the messages in batch and invoke Lambda functions to process them.

We will also use Amazon SNS Filter feature when sending messages to SQS queues. Because in distributed architectures , not every message is required to be sent to every downstream microservices. So we will conditionally sent messages based on some attributes in the message body. We will use filter criteria's in SNS filter features before sending messages to the SQS queues. So in this article, we will learn how fan out patterns apply with using SNS and SQS that enable asynchronous message communication when building distributed microservices architectures.

As you know that, when we implement any architecture on AWS, we have 2 main steps;

1. Create infrastructure on AWS Cloud
2. Develop Lambda code for interacting SQS

So we will start with the first step, Lets Create this architecture infrastructure on AWS Cloud. But before lets start to remember pub/sub fan out and Topic Queue Channing Patterns and best practices.

Pub/Sub Fan-Out and Topic Queue Channing Patterns with SNS, SQS and Lambda

Lets start with What is Fan out ? Fan-out is a messaging pattern where piece of message is distributed or ‘fanned out’ to multiple destination in parallel. The main idea is each of destinations can work and process this messages in parallel.

One way to implement this messaging pattern is to use publisher/subscriber or pub/sub model. In the pub/sub model we define a topic which is logical access point to enabling message communication with asynchronously.

Workshop Studio

A publisher simply sends the message to the topic. After that this message is immediately fanned out to all subscribers of this topic. This message communication is completely decoupled and asnycronously. Each service can operate and scale independently and individually without having any dependency of other services. The publisher doesn’t need to know who is consuming this message that is broadcasting. And the subscribers don’t need to know where the message comes from. The best way to build pub/sub fan out messaging on AWS is to use Amazon SNS. Amazon SNS is fully managed reliable and secure pub/sub messaging service.

So this architectural challenges recommends by using messaging patterns, resulting in loosely coupled communication between highly cohesive components to manage complexity in serverless architectures. A common approach when one component wishes to deliver the same message to multiple receivers is to use the fanout publish/subscribe messaging pattern.

Publish/Subscribe Messaging Pattern

What is Pub/Sub Messaging? Publish/subscribe messaging, or pub/sub messaging, is a form of asynchronous service-to-service communication used in serverless and microservices architectures. In a pub/sub model, any message published to a topic is immediately received by all of the subscribers to the topic.

Pub/sub messaging can be used to enable event-driven architectures, or to decouple applications in order to increase performance, reliability and scalability. The Publish Subscribe model allows messages to be broadcast to different parts of a system asynchronously.

Topic-Queue Chaining & Load Balancing Pattern

We can use a queue that acts as a buffer between the service from which it was called from asynchronous invocations. By this way we can avoid loss data if the service to fail or the task to time out. This can help minimize the impact of peaks in demand on availability and responsiveness for the consumer microservice.

Workshop Studio

If we look at the Topic-Queue Chaining pattern, — you can see picture in the slide, There are 3 subscriber backend services;

1. Customer Notification Services which’s interested in getting notified from publisher microservices.
2. Customer Accounting Services
3. Extraordinary ride Service

So if one of these services can be down or getting exception or taken offline for maintenance, then the events will be loses, disappeared and can’t process after the subscriber service is up and running. A good pattern to apply here is topic-queue-chaining. That means that you add a queue, in our case an Amazon SQS queue, between the Amazon EventBridge and each of the subscriber services. This is enough for learning Serverless Architecture Patterns and Best practices.

Now I go back to our main topic which is Hands-on Labs: Fan-Out Serverless Architectures Using SNS, SQS and Lambda.

Create Infrastructure for Pub/Sub Fan-Out Architecture with SNS, SQS and Lambda

We are going to Create Infrastructure for Pub/Sub Fan-Out Architecture with SNS, SQS and Lambda. Here you can see the overall architecture that we are going to build in this hands-on section.

Publish/Subscribe Fan-Out Pattern in Serverless Architectures Using SNS, SQS and Lambda

Create Order Acknowledgment Microservice Lambda Function

Goto AWS Management Console and follow the steps below;

• Create Lambda function — OrderAcknowledgement
• Permissions — Create a new role from AWS policy templates — Simple microservice permissions — DynamoDB — Amazon SNS publish policy — SNS

This is so important feature that we can set required permissions when creating lambda function with understandable names. Check architecture diagram and See arrows from lambda to other services — DynamoDB, SNS
and search for names and see actions : SNS publish.

According to our architecture we publish SNS message from our Order Acknowledgment microservices, so it needs to authorization of SNS:Publish action, otherwise it getting error.

Basically we should think lambda interactions and give required permissions of execution roles.

• Order Acknowledgment microservices
  SNS:Publish
  DynamoDB:Put
• Inventory microservice
  DynamoDB:Put

Create Lambda Function with these configurations.

Create API Gateway

• REST API — build
• OrderAPI — regional — default settings
• Actions — Create Resource — order
• Create Method — POST — Check — Use Lambda Proxy integration
• Lambda Function — OrderAcknowledgement
• Create API Gateway
• Deploy API — new stage — prod — deploy

Open postman send post request

• Invoke URL: https://qgww7llfgl.execute-api.us-east-2.amazonaws.com/prod/order

{
 “statusCode”: 200,
 “body”: “\”Hello from Lambda!\””
}

See it worked.

Create DynamoDB tables

• order — PK — id — inventory — PK — code
• default settings and create table

Create Downstream microservices

• notification microservices
• inventory microservices
• shipment microservices

Basically we should think lambda interactions and give required permissions of execution roles.

• goto lambda — execution role — add policy
• inventory microservice
  DynamoDB:Put
  sqs:poller
• notification-queue
  inventory-queue
  shipment-queue
  sqs:poller

Create SQS queues

• notification-queue, inventory-queue, shipment-queue
• standart default created

Create Event Source Mapping Polling Invocation SQS and Lambdas

• goto lambda functions
• add trigger — sqs
• notification->queue, inventory->queue, shipment->queue

As you can see that, we have successfully created lambda functions and sqs queues, in the next part, we are continue to create SNS topic.

Create SNS Topic

• Create topic — standart — OrderTopic
• create SNS Topic

Create subscriptions SNS — SQS subs

• protocol — SQS, inventory-queue, shipment-queue
• Create subscription

Amazon SNS works closely with Amazon SQS. When you subscribe an Amazon SQS queue to an Amazon SNS topic, you can publish a message to the topic and Amazon SNS sends an Amazon SQS message to the subscribed queue.

Develop Order Acknowledgment Microservices Lambda Function Code

Now we are starting to second part of serverless project which is Development part. We are going to Develop Order Acknowledgment Microservices Lambda Function Code.

Publish/Subscribe Fan-Out Pattern in Serverless Architectures Using SNS, SQS and Lambda

Lets Create NodeJS Project with SNS SDK Packages for Lambda function. Open VS Code, create “fanout-project” folder. This will be our NodeJS project for this section. Locate folder and run npm command in order to create package.json:

npm init -y

See and edit default package.json is created. The first thing we should do
Add “type”: “module”:

package.json 
 “keywords”: [],
 “author”: “”,
 “license”: “ISC”,
 “type”: “module” — — — ADDED
}

this provide to use ES6 codes into nodejs projects. for example “import statements”.

Install required SQS sdk packages

Think about interaction into Order Ack Lambda Function. Basically we need to generate id with uuid library, we need to save into DynamoDB, we need to publish message to SNS. So our required packages are uuid, dynamodb-client, sns-client.

Run command on package.json level:

npm install uuid
npm install @aws-sdk/client-sns
npm install @aws-sdk/client-dynamodb

See package installed in package.json file:

{
 “name”: “section8”,
 “version”: “1.0.0”,
 “description”: “”,
 “main”: “index.js”,
 “scripts”: {
 “test”: “echo \”Error: no test specified\” && exit 1"
 },
 “keywords”: [],
 “author”: “”,
 “license”: “ISC”,
 “type”: “module”,
  "dependencies": {
    "@aws-sdk/client-dynamodb": "^3.105.0",
    "@aws-sdk/client-sns": "^3.105.0",
    "@aws-sdk/util-dynamodb": "^3.105.0",
    "uuid": "^8.3.2"
  }
}

Now its time to develop our Lambda function code. We should plan our developments for Order Acknowledgment Microservices. Lets write pseo code first:
// 1- get request body payload
// 2- generate id with uuid library and prepare payload
// 3- publish message to sns topic with using sns sdk package
// 4- save order item into dynamodb with using dnamodb sdk package
// 5- return back snyc order payload with generated id to the api gateway

Develop Order Acknowledgment Microservices Lambda Function Code

In 3–4 steps we will Publish Message to Topic on Amazon SNS using AWS SDK. For that purpose we will use PublishCommand and snsClient.js files to communicate with SNS from Lambda function.

First thing to create client nodejs module to interact with SNS APIs.

create — snsClient.js

import { SNSClient } from “@aws-sdk/client-sns”;
// Set the AWS Region.
const REGION = “us-east-2”;
// Create SNS service object.
const snsClient = new SNSClient({ region: REGION });
export { snsClient };

Now lets develop our actual lambda code with following the our pseo codes:

import { v4 as uuidv4 } from ‘uuid’;
import {PublishCommand } from “@aws-sdk/client-sns”;
import {snsClient } from “./snsClient.js”;
export const handler = async (event) => {
 console.log(“event:”, JSON.stringify(event, undefined, 2));
// pseo code
 // 1- get request body payload
 // 2- generate id with uuid library and prepare payload 
 // 3- publish message to sns topic with using sns sdk package
 // 4- save order item into dynamodb with using dnamodb sdk package
 // 5- return back snyc order payload with generated id to the api gateway 
 
 try {
 
 // validations
 if (event.httpMethod != ‘POST’) {
 throw new Error(`Http Method should be POST. You are requesting : “${event.httpMethod}”`);
 }
// 1- get request body payload
 // expected request payload : { userName : swn, attributes[firstName, lastName, email ..] 
 const orderRequest = JSON.parse(event.body);
 if (orderRequest == null || orderRequest.type == null) {
 throw new Error(`order type should exist in orderRequest: “${orderRequest}”`);
 }
// 2- generate id with uuid library 
 const orderId = uuidv4();
 orderRequest.id = orderId;
// 3- publish message to sns topic with using sns sdk package 
 let params = {
 Message: JSON.stringify(orderRequest),
 TopicArn: process.env.TOPIC_ARN,
 }; 
 const data = await snsClient.send(new PublishCommand(params));
 console.log(“Successfully published SNS Message.”, data);
 
 // 4- save order item into dynamodb with using dnamodb sdk package
// 5- return back snyc order payload with generated id to the api gateway 
 return {
 statusCode: 200,
 body: JSON.stringify({
 message: `Successfully finished order create operation: “${JSON.stringify(orderRequest)}”`,
 body: data
 })
 };
} catch (e) {
 console.error(e);
 return {
 statusCode: 500,
 body: JSON.stringify({
 message: “Failed to perform operation.”,
 errorMsg: e.message,
 errorStack: e.stack,
 })
 };
 } 
};

As you can see that we have Publish Message to Topic on Amazon SNS using AWS SDK, and finished development of Order Ack Microservices.

Deploy and Test Order Acknowledgment Microservices Lambda

we are going to Deploy and Test Order Acknowledgment Microservices Lambda.

Publish/Subscribe Fan-Out Pattern in Serverless Architectures Using SNS, SQS and Lambda

• goto index.js path — ctrl+a select all — right click — compress — index.zip
• goto OrderAck lambda function — upload zip — success

Test with Postman

We will send request to API Gateway that trigger to our Lambda function.

• Postman Invoke URL: https://46j7y7zw78.execute-api.us-east-2.amazonaws.com/default/OrderAcknowledgement

Expected payload:
 {
 “item”:”iphone”,
 “status”:”CREATED”,
 “type”:”SHIP_REQUIRED”
 }
RESPONSE:
{
 “message”: “Successfully finished order create operation: \”{\”item\”:\”iphone\”,\”status\”:\”CREATED\”,\”type\”:\”SHIP_REQUIRED\”,\”id\”:\”6294baa8–23a9–47f5-b5fd-a7c647b2e824\”}\””,
 “body”: {
 “$metadata”: {
 “httpStatusCode”: 200,
 “requestId”: “071e3f09–36ad-5004-a646–6410678004a7”,
 “attempts”: 1,
 “totalRetryDelay”: 0
 },
 “MessageId”: “ee360a8e-39be-55b1–8496–519768936112”
 }
}

As you can see that we have successfully got response and execute our E2E use case which is fan-out pattern with using Lambda, SNS and SQS.

Message Filter Pattern in Amazon SNS

By default, an Amazon SNS topic subscriber receives every message published to the topic. To receive a subset of the messages, a subscriber must assign a filter policy to the topic subscription.
When you publish a message to a topic, Amazon SNS compares the message attributes to the attributes in the filter policy for each of the topic’s subscriptions. If any of the attributes match, Amazon SNS sends the message to the subscriber. Otherwise, Amazon SNS skips the subscriber without sending the message. If a subscription doesn’t have a filter policy, the subscription receives every message published to its topic.

As you can see that we have successfully developed Hands-on Lab : Publish/Subscribe Fan-Out Pattern in Serverless Architectures Using SNS, SQS and Lambda. To see full developments of this hands-on lab, you can check below course on Udemy.

Step by Step Design AWS Architectures w/ Course

I have just published a new course — AWS Lambda & Serverless — Developer Guide with Hands-on Labs.

In this course, we will learn almost all the AWS Serverless Services with all aspects. We are going to build serverless applications with using AWS Lambda, Amazon API Gateway, Amazon DynamoDB, Amazon Cognito, Amazon S3, Amazon SNS, Amazon SQS, Amazon EventBridge, AWS Step Functions, DynamoDB and Kinesis Streams. This course will be 100% hands-on, and you will be developing a real-world application with hands-on labs together and step by step.

Source Code

Get the Source Code from Serverless Microservices GitHub — Clone or fork this repository, if you like don’t forget the star. If you find or ask anything you can directly open issue on repository.