Serverless Application for Anomaly Detection Using Sigma

Serverless application development is currently being one of the topmost trending technologies. Although that concept has a great potential to redefine the future of application development, serverless application development cycle was never been easier. With the recent release of Sigma IDE from SLAppForge, now serverless application development becomes much easier because it streamlines the design, development and deployment of serverless applications, concealing the underlying platform-level complexities from developers, which has slowed down the widespread adoption of serverless computing.

In this blog post, I’m going to explain how to use Sigma IDE to implement real-world use case for serverless application development.

Use case description

ReserveNow.LK hotel booking service provider which manages an application to handle hotel bookings which are purchased by their client hotels. They currently have a requirement to process booking requests asynchronously for further analyze request patterns and failures to enhance their service in the future. Since they don’t want to introduce any delay to booking request handing process due to this requirement, they have decided to push these booking messages to a message queue.

Our exercise here is to implement this asynchronous booking request processing application using AWS services. To make it simple, let’s focus only two main requirements in this case.

  1. The application should identify booking request anomalies based on the booking dates and detected anomalies should be persisted to a NoSQL database for further analysis using some data analytical tool.
  2. It should detect booking failures and notify failure state to the Ops team via an SMS message

Selected AWS services

  • Since we need to decouple booking handling and asynchronous booking request processing, we need a queueing system in-between. AWS Simple Queue Service (SQS) is the best solution for this. So booking engine will publish messages to the SQS, and our application will poll messages from SQS and do required post-processing for further analysis.
  • Detected anomalies should be persisted to a NoSQL database which obviously forces us to choose DynamoDB for that.
  • For notifying Ops team via an SMS about the booking failures, we can use AWS Simple Notification Service(SNS).

Those are the service components that we can use for each capability that we required. However, there is one big missing part which is the processing part or the FaaS component of the application to detect anomalies and failures of the booking requests. When we are talking about the serverless and FaaS with AWS, the only and the best option for that is AWS Lambda.

Now all the components are selected, but there is one more thing. Based on the architectural behavior, AWS SQS couldn’t use as a trigger for AWS lambda to process messages from an SQS queue. If we want to process SQS messages via a Lambda, we need to trigger the Lambda function periodically via a CloudWatch rule and then let Lambda fetch messages from the configured SQS queue.

So overall architectural view of the application would be like this,

Implementation

Here comes the most interesting part !!! Before continuing further, I would like to invite you to register for Sigma IDE which is completely free —all you required is a valid email address.

For testing purposes, Sigma provides you, a test AWS account as well — of course with limited billing value ;-). So other than the Sigma registration, now you only need a GitHub account.

Lambda Function Flow

Lambda function execution flow should be like this for facilitating our requirement.,

  • Fetch messages from the queue
  • Go through each message and check for booking anomalies. Booking anomalies will be detected on two rules, if booking start date is 6 months after the current date or booking period (the difference between booking end date and start date) is greater than 20 days, it will be detected as booking anomaly. See following sample booking request to understand the structure of the message
{
"resellerId": "81023I",
"bookingRef": "SL-booking-NR-1284404",
"bookingReqProcessingState": "Success",
"bookingRequest": {
"startDate": "2018-11-19",
"endDate": "2018-12-02",
"pax": "2",
"city": "PAR",
"grade": "NR"
}
}
  • Detected booking anomalies should be persisted to the DynamoDB database.
  • If there are booking failures (based on the value of the “bookingReqProcessingState” field value), those booking failures should be notified to the users via an SMS message (to be simple, let’s send an SMS message only for one number)
  • Delete all processed messages from the SQS queue (SQS won’t deleted messages automatically once you read the messages. You have to specifically delete messages once you complete the processing for that message)

Implementing Lambda Logic

Once you register to Sigma you can create a project by providing appropriate name, and region that you want to work with. After project creation, you should be navigated to the IDE view which is like below.

Add data time processing utility dependency — In this example, I’m going to use few date time parsing and formatting operations, which will require external / 3rd party dependency called “date-and-time” node module. You can easily add that by clicking on the small plus button in the left corner of the header toolbar. Once you click on that button you will get an option to search for your node module and then you can add it as required.

Trigger lambda periodically — to trigger our lambda periodically and fetch messages from SQS, we need a CloudWatch rule. For that, you have to drag and drop the CloudWatch resource for the variable event in the handler function (highlighted in red) like below.

Once you drop it, you should get a popup to configure your CloudWatch rule. Give some meaningful name for the rule and select the schedule tab and configure the rate for 5 minutes as below to make sure our lambda function will be triggered in every five minutes.

Fetch messages from SQS — to fetch messages from SQS, drag and drop the SQS resource from the left panel to the body of the handler method — of course before the last callback handling line. Once you drop the SQS resource, it should give you a popup to configure your SQS queue. Assuming that, we are using a new queue (make sure New Queue tab selected), configure queue name, change the operation to “Get and Remove Message”and Maximum Number of Messages to 10 — this is the max value which can be fetched for a single attempt from the SQS side; like below,

After that your IDE and Lambda file should be like this,

Now you can code your custom logic within the call back functions — as described in above comments. First call back function is supposed to process received messages and filter out any required messages from the receivedMessages list if you want to stop removing any messages from that list. Once it goes out from the first call back function, all the messages in the receivedMessages list will be deleted. For this use case, it’s required to process all the receivedMessages and delete them after completion of the processing logic. For my use case, the first call back function should be coded like below,

There are two remaining parts in the above code, which are supposed to integrate with SNS to send SMS notification messages about the detected failure and to integrate with DynamoDB to persist the detected Anomaly.

Make sure you have imported date-and-time module with the name date, at the top of your lambda function like this “let date = require(‘date-and-time’);”

Integrate with SNS —we have to integrate our lambda with SNS to send SMS notification messages for detected booking failures. For that, drag and drop an SNS resource after the notificaitonMsg variable declaration line. Configure SNS resource as below,

Then configure the generated then and catch callback functions some log message messages like below,

Integrate with DynamoDB — we need to integrate with DynamoDB and persist detected anomalies. For that drag and drop a DynamoDB resource from the resource panel to the place that we need to add code block for persisting detected anomalies — it should be within second if block after calculating the insertTimeStr variable. Configure the DynamoDB resource by filling its popup as below,

Once you click the Inject button you should get a code block like this for DynamoDB — change the then and catch callbacks with appropriate logic as shown below,

Now we are done with coding. Final source code should be like this,

Commit, Build and Deploy

Once you are done with code-level changes, you just have to click on the Deploy button from the header toolbar. It will commit your source code to your GitHub account and build the project and finally deploy the project on AWS. While deploying it will create all the required resources on the AWS side and configure all the required permissions and roles for your Lambda function execution.

Voila!! Now we are done… As you have experienced, the only thing that you have to focus on is the implementation of your custom Lambda processing logic. Resource management, permission management, resource bindings and deployments of your Lambda function will be taken care of the Sigma IDE for you — that’s the specialty of it which streams line resource management and deployment to facilitate a hassle-free serverless application development experience for its users.

Testing

Currently, Sigma IDE is in the beta version which doesn’t support testing of functionality for you. However, very soon you will be able to experience it. For now, you have to go to AWS services and go to the Simple Queue Service to push a message to our input queue — anomalyDectionQueue. Send few sample messages (using the above format with different values) and experience the SMS notifications and anomaly persistence behavior.


I have written another blog to implement the same application without using Sigma IDE. You can easily experience the difference between directly using AWS resources vs using Sigma IDE for implementing your serverless application.

Sigma IDE is currently in beta version, we mostly welcome your suggestions and feedback to improve its capabilities and features. Feel free to contact us either at Sigma IDE, Report Issue option or from here.


Call To Action

  • Clap. Appreciate and let others find this article.
  • Comment. Share your views on this article.
  • Follow me. Manjula Piyumal to receive updates on articles like this.
  • Keep in touch. LinkedIn, Twitter