Building Fan-Out Serverless Architectures Using SNS, SQS and Lambda (Event Driven Architecture)
Are you excited to explore event-driven architecture? Absolutely, let’s get started! I believe that solving real-world problems is an excellent way to learn about technologies and system architecture. Therefore, we’ll approach Fan-Out Serverless Architecture in a similar manner. So, let’s begin building a hotel room reservation system.
I have a beginner friendly step-by-step tutorial to help you make your first fan-out architecture project.
Challenges in Current Infrastructure
In the hotel industry, managing room bookings efficiently while ensuring seamless communication between various systems poses a significant challenge.
Traditional point to point communication methods lead to tightly coupled systems, hindering scalability and flexibility. Moreover, handling asynchronous message communication manually introduces complexity and potential points of failure, impacting the customer. To address this I am proposing a “Hotel Room Booking System” in an event driven fanout pattern. This solution will assure Scalability, Decoupling, Reliability and easy Maintenance by incorporating various cloud based managed services provided by AWS.
The current room booking system faces several challenges:
Dependency Management: Tight coupling between booking management, payment processing, room availability, and customer notifications leads to complex dependencies.
Scalability: With fluctuating demand and varying booking volumes, the existing system struggles to handle peak loads efficiently.
Reliability: Manual handling of asynchronous communication introduces reliability issues, such as message loss or duplication.
Maintenance: Since the systems are deployed on premise, hence we are also responsible for its end-to-end maintenance.
The proposed solution offers the following benefits:
Scalability: By leveraging SNS, the system gains the ability to scale seamlessly to handle fluctuating booking volumes.
Decoupling: Implementing the fanout pattern with SNS enables decoupling of booking management, customer notifications and Analytics services.
Reliability: SNS provides built-in features for handling message delivery, including failure handling and retry logic. This ensures reliable communication between system components, reducing the risk of message loss or duplication.
Maintenance: Since the services used to architect the system are fully managed services by AWS, hence AWS is responsible for most of the heavy lifting tasks such as provisioning, scaling and patch management.
Proposed Architecture:
PART 1) Creating Lambda functions
We have three lambda functions, all these lambda functions are running Node.js 20.x
- Create a new Lambda function: Click on the “Create function” button.
- Choose Author from scratch: Select the “Author from scratch” option to create your function from scratch.
- Provide function details:
- Function name: Enter “reservationprocessing” as the function name.
- Runtime: Choose Node.js,
- Permissions: Give permissions to access DynamoDB and SNS, or create a new role with necessary permissions.
- Click on Create function: This will create the Lambda function with basic configurations.
- Save your changes: After attaching the necessary policies, go back to the Lambda function console.
- Write your Lambda function code: In the Lambda function console, scroll down to the function code section and write your Lambda function code in the editor provided.
- Similarly, we made two other microservices namely notificationMicroservice and inventoryMicroservice and below is the code.
- “reservationProcessing”: intended for saving the reservation details to the “reservation” (DynamoDB database) and publishing the reservation details to the SNS “reservationTopic”.
// index.js (reservationProcessing lambda)
import { v4 as uuidv4 } from 'uuid';
import {PublishCommand } from "@aws-sdk/client-sns";
import {snsClient } from "./snsClient.js";
import { PutItemCommand } from "@aws-sdk/client-dynamodb";
import { marshall, unmarshall } from "@aws-sdk/util-dynamodb";
import { ddbClient } from "./ddbClient.js";
export const handler = async (event) => {
try {
if (event.httpMethod != 'POST') {
throw new Error(`Only Http Method POST. allowed : "${event.httpMethod}"`);
}
const roomOrderRequest = JSON.parse(event.body);
if (roomOrderRequest == null || roomOrderRequest.type == null) {
throw new Error(`order type should exist in roomOrderRequest: "${roomOrderRequest}"`);
}
const orderId = uuidv4();
roomOrderRequest.id = orderId;
let params = {
Message: JSON.stringify(roomOrderRequest),
TopicArn: "arn:aws:sns:us-east-1:000000000:ReservationTopic",
};
const data = await snsClient.send(new PublishCommand(params));
const dynamodbParams = {
TableName: "reservation",
Item: marshall(roomOrderRequest || {})
};
const createResult = await ddbClient.send(new PutItemCommand(dynamodbParams));
return {
statusCode: 200,
body: JSON.stringify({
message: `Successfully finished order create operation: "${roomOrderRequest}"`,
body: data
})
};
} catch (e) {
console.error(e);
return {
statusCode: 500,
body: JSON.stringify({
message: "Failed to perform operation.",
errorMsg: e.message,
errorStack: e.stack,
})
};
}
};
// ddbClient.js
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
const REGION = "us-east-1";
const ddbClient = new DynamoDBClient({ region: REGION });
export { ddbClient };// snsClient.js
import { SNSClient } from "@aws-sdk/client-sns";
const REGION = "us-east-1";
const snsClient = new SNSClient({ region: REGION });
export { snsClient };2. “notificationMicroservice”: intended to send the notifications to customer regarding booking status.
3. “inventoryMicroservice”: intended to maintain the inventory of rooms. Saves the booking in “inventory” database.
//index.js (inventoryMicroservice)
import { PutItemCommand } from "@aws-sdk/client-dynamodb";
import { marshall, unmarshall } from "@aws-sdk/util-dynamodb";
import { ddbClient } from "./ddbClient.js";
export const handler = async function(event) {
try {
for(const record of event.Records) {
const snsPublishedMessage = JSON.parse(record.body);
const reservationRequest = JSON.parse(snsPublishedMessage.Message);
if (reservationRequest == null || reservationRequest.type == null ) {
throw new Error(`order type should exist and should be SHIP_REQUIRED in reservationRequest: "${reservationRequest}"`);
}
reservationRequest.code = reservationRequest.item;
const dynamodbParams = {
TableName: 'inventory',
Item: marshall(reservationRequest || {})
};
const createResult = await ddbClient.send(new PutItemCommand(dynamodbParams));
console.log("Successfully create item into order table.", createResult);
}
} catch (e) {
console.error(e);
}
};// ddbClient.js
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
const REGION = "us-east-1";
const ddbClient = new DynamoDBClient({ region: REGION });
export { ddbClient };PART 2) Creating API Gateway
Steps to create API gateway, To create a POST route “/reservation” in AWS API Gateway, we followed these steps:
- Navigate to AWS API Gateway Console: Log in to your AWS Management Console, go to the API Gateway service.
- Create a New API: click on “Create API” and choose the REST API type.
- Define Resources and Methods:
- Click on “Create Resource” to define a resource (/reservation). After creating the resource, click on it to define the method (POST in this case).
- Set up Integration: After defining the method, you need to set up the integration. We can choose to integrate with Lambda Function, (reservationProcessing) HTTP endpoint, Mock integration, etc. If we’re integrating with a Lambda function, select “Lambda Function” and choose “reservationProcessing” lambda function.
- Set up Method Request: Configure the request parameters in the Method Request section. For a POST request, we might have request body parameters.
- Deploy API: Once you have configured your API, deploy it to a stage (prod). This will generate an endpoint URL that you can use to make requests to your API.
PART 3) Creating SQS queue
In the AWS console search for SQS.
- Create a New Queue:
- Click on the “Create Queue” button select “Standard Queue”.
- Enter the Queue Name (“notification-queue”).
- Click on the “Create Queue” button to create the queue.
2. Repeat the above steps for the other two queues: Create “inventory-queue” and “analytics-queue” using the same process.
3. Configure Queue Permissions: You can do this in the “Permissions” tab of each queue. In the Access policy, you need to allow the queue to receive messages from SNS topics(we will implement SNS topics and subscription the next section) and attach lambda triggers. So we will revisit permissions for SQS.(see PART 5)
PART 4) Creating SNS topic and subscriptions
Now lets create SNS topic and make subscriptions :
Search SNS in AWS console.
- Create a New Topic:
- Click on the “Create topic” button.
- Enter the Topic Name ( “reservationTopic”).
- Click on the “Create topic” button to create the topic.
2. Subscribe Queues to the Topic:
- After creating the topic, click on it to open its details.
- Click on the “Create subscription” button.
- Select the protocol as “Amazon SQS” for the first three subscriptions.
- Choose the respective queues for the subscriptions (“notification-queue”, “inventory-queue”, “analytics-queue”).
- Click on the “Create subscription” button to create each subscription.
3. Subscribe Email Addresses:
- Click on the “Create subscription” button again.
- Select the protocol as “Email” for the next two subscriptions.
- Enter the email addresses for the subscriptions (“hotelpartner@gmail.com”, “taxipartner@gmail.com”). hotelpartner@gmail.com is the email address of the hotel for which the user has booked the room and taxipartner@gmail.com is the email for the taxi company.
- Confirm the subscriptions by clicking on the confirmation link sent to the email addresses.
4. Set Subscription Filter Policy:
- Since we only want to notify “taxipartner@gmail.com” if and only of the user booked for the “room reservation+taxi”. If the user has selected only “room reservation” without taxi then we will not notify taxi company.
- Apply subscription filter policy for “taxipartner@gmail.com”. This filter is of type Message body. and will only be triggered if the body of incoming message from SNS topic has “type”:”ROOM_TAXI”.
PART 5) Giving permissions to SQS (PART 3 continue)
All the SQS queues should have permissions to receive data from SNS topic and a lambda trigger.
// Sample Access policy for SQS (DO NOT PASTE THIS)
// Id , sid and ARN will be different
{
"Version": "2012-10-17",
"Id": "Policy1799962918573",
"Statement": [
{
"Sid": "Stmt1710002900000",
"Effect": "Allow",
"Principal": "*",
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:us-east-2:123456789012:inventory-queue",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:sns:us-east-2:123456789012:ReservationTopic"
}
}
}
]
}Repeat similar steps for “notification-queue”
PART 6) Creating DynamoDB tables
To create two DynamoDB tables named “inventory” and “reservation,” follow these steps:
Search for DynamoDB in the AWS console and select DynamoDB.
- Create the “inventory” Table:
- Click on the “Create table” button.
- Enter “inventory” as the Table name.
- Specify the Primary key:
- Partition key: Enter the name for your partition key ( “code”).
- Choose the data type for the partition key ( “String”).
- Configure additional settings as needed, such as Provisioned or On-demand capacity, and other options.
- Click on the “Create” button to create the table.
2. Create the “reservation” Table:
- Follow the same steps as above, but this time enter “reservation” as the Table name.
- Specify the Primary key:
- Partition key: Enter the name for your partition key (“id”).
- Choose the data type for the partition key (“String”).
- Configure additional settings as needed.
- Click on the “Create” button to create the table.
PART 7) Bonus (you can use postman). Creating a front-end application
Here I am utilizing a free react template from mui.com(great website for personal projects). Download the source code and Tweak it to your needs.
Template used: https://mui.com/store/items/minimal-dashboard-free/
Other templates: https://mui.com/store/?utm_source=marketing&utm_medium=referral&utm_campaign=templates-cta#populars
Follow a very simple process of hosting this website on a S3 bucket(Static hosting).
Now, your React.js webapp is hosted, access it using the hosted link provided under “static hosting” option under “Properties”.
There can be 2 scenarios:
Scenario 1: General Reservation
- When user click on “Select” we will trigger general reservation (i.e. without taxi service). Only hotel partner will receive email notification.
- I am using this React.js code to do a post call on “/reservation” with the provided payload(“type”: “ROOM").
- Booking details will be save to both reservation and inventory tables.
// post call for general reservation
const postDataToServer = async (data) => {
try {
const response = await fetch('https://aabbccddeeffgg.amazonaws.com/prod/reservation', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
"user_name": user_name,
"user_email": user_email,
"user_phone": user_phone,
"user_address": user_address,
"user_city": user_city,
"user_state": user_state,
"user_zip": user_zip,
"user_country": user_country,
"item": data.name,
"price": data.price,
"date_from": dateFrom,
"date_to": dateTo,
"status": "CREATED",
"type": "ROOM"
})
});
if (!response.ok) {
throw new Error('Network response was not ok');
}
} catch (error) {setOpen(true);
console.error('Error:', error);
}
};
Scenario 2: Room Reservation with Taxi Service
- Room reservation with taxi: when user selects “Select+ Taxi Service”. In this case both the hotel partner and taxi company will receive booking details by email(SNS subscription filter policy in action).
- I am using this React.js code to do a post call on “/reservation” with the provided payload(“type”: “ROOM_TAXI”).
- Booking details will be save to both reservation and inventory tables.
const postDataToServerTaxi = async (data) => {
try {
const response = await fetch('https://aabbccddee.amazonaws.com/prod/reservation', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
"user_name": user_name,
"user_email": user_email,
"user_phone": user_phone,
"user_address": user_address,
"user_city": user_city,
"user_state": user_state,
"user_zip": user_zip,
"user_country": user_country,
"item": data.name,
"price": data.price,
"status": "CREATED",
"type": "ROOM_TAXI"
})
});
if (!response.ok) {
throw new Error('Network response was not ok');
}
console.log(data);
} catch (error) {setOpen(true);
console.error('Error:', error);
}
};
I have used https://lucid.co/ to create my architectural diagrams. I love making my aws architectural diagrams using lucid since you can start from “free subscription” and it is very easy to make.
As you may have noticed, specific implementation details for the notificationMicroservice and analyticsMicroservice haven't been provided, as they can be tailored to meet the user's specific requirements.
An example implementation for the notificationMicroservice could involve a basic task, such as sending an email to the user, with the email address derived from the data received from the notification queue.
For the analyticsMicroservice, a variety of AWS services can be employed. One approach is to leverage multiple AWS services for data processing and analysis. Please share your thoughts or preferences on the implementation.
Thanks for reading, happy coding!
