Triggering Lambda function from SQS and store the message in S3 bucket with Terraform(IAC)

Introduction

Amazon SQS is a reliable, easy to manage, scalable queuing service. SQS is a simple and cost effective cloud application.

AWS SQS can handle any amount of data at any throughput level without losing messages.
It does not prevent the other services from running indefinitely.

By scaling high-availability messaging clusters, SQS helps to reduce admirative task. While we only pay for what we consume. AWS SQS enables us to save critical data that would otherwise be lost if the entire application or any component failed.

Essentially, the SQS queue serves as a buffer between the application components that receive data and the other parts of the system that process the data.

Message-oriented architecture-based applications use SQS.
When the processing servers are unable to process the work quickly enough (for whatever reason), the work is queued so that the processing servers can work on it when they have available resources to process the request.
This means that no work is lost due to a lack of resources.

Amazon SQS makes certain that every message is delivered at least once.

Queues are classified into two types :

Architecture Diagram :

A Basic Use Case :

Consider a large flash sale on E-commerce where people buy and sell a wide range of products.. The requirement now is that all buyer and seller requests be processed in the order that they were received in the queue. To meet this requirement, we’ll use the FIFO queue to process transactions in a single flow.

A mobile company is holding a flash sale for their new model with great features at the best price. It is expected that a huge number of buyers will place their orders. The company is holding limited stock for a limited period, so it’s important to track the order that arrived first. Your flash sale receives huge responses and only the buyers who place the order first will receive the product and remaining users get to try in the next sale. Once the request is received, they are sent to a FIFO queue before they are processed.

Let’s understand how messages get in and out of the queue. Assume the consumer asks for a batch of up to 300 messages, AWS SQS starts filling the batch with the oldest message (REQ A1). Now SQS keeps filling the queue until the batch is full. In our case, assume batch contains only three requests and now the Queue is empty. Once the message batch has left from the queue SQS considers the batch to be In-flight until the batch gets processed completely and deletes or the visibility timeout gets expires.

This is simple to handle when you only have one consumer. The consumer receives a batch of messages, processes them, and then deletes them. The consumer is now prepared to process and accept the next batch of messages. You can also add an auto scaling group to scale your processing power based on your needs.
Please keep in mind that SQS will not release the next batch of messages until the previous batch has been deleted.

In this article I’ll try to focus on the Standard Queue and i will provision the following resources:

• S3 Bucket
• IAM role for Lambda
• AWS Lambda Function
• SQS Queue
• Lambda trigger

Prerequisites:

A code editor (Vscode, Atom, etc), install Terraform plugin extension to Visual Studio Code, have an AWS account and an AWS IAM user with the relevant permissions. Depending on your operating system, install AWS CLI.

Config files and Python code for Lambda function

Go to my repo: GitHub= https://bit.ly/3p2zEPW

Clone my repo to your code editor using git clone command below and follow the instruction in (README.md) file.

git clone https://github.com/Assassin010/Triggering-Lambda-function-from-SQS-S3

Happy coding!!!

END — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

If you like my work and want to support me…

The BEST way is following me on Medium here.

Feel free to give claps, or writing comments so I know how helpful this post was for you.

Gauthier Kwatatshey__

Connect with me on LinkdIn : Gauthier Kwatatshey | LinkedIn

#Devops #aws #iac #terraform #python