Implementing a Serverless Batch File Processing Application (Part 2) — The Sigma Way
In my previous article, we discussed how to build a fully functional batch file processing solution with serverless components and services offered by AWS. Here is the high-level architecture diagram of our application, just to refresh your memory.
The journey we had …
When we were building this application, we had to face some difficulties as well. First we had to take an arduous journey through several AWS service consoles to configure the required components of our application such as a SNS topic, a S3 bucket, a Lambda function as well as a CloudWatch trigger. Then we had to go though the AWS SDK documentation to learn how to integrate these components within lambda code and also to figure out which execution permissions are required by the Lambda to access these components correctly.
These steps seems a bit too complex and tiresome for most of us. Specially if you are someone new to serverless technologies or someone who are not so familiar with AWS service consoles, you can easily get lost in the process. So is there any simpler way to get this done?
The Sigma way …
We at SLAppForge had the same thoughts when we were working on AWS for some other projects, and finally decided to make it simpler, not only for us, but also for everyone. As a result, Sigma, a browser based serverless development environment saw the light of day!
So let’s see how we can build the same solution using Sigma in much less time and effort. For that first you need to sign up to Sigma (which is completely free! 😀) and create a project. And don’t forget to provide a valid access key and secret key to your AWS account so Sigma can deploy the solution for you.
You can choose whether Sigma should store your keys for future use or not. Even if you chose to store them, they will be stored in encrypted form so that no one but only you can retrieve them back.
Once you create the project, let’s start adding the necessary components and code. Unlike in the previous way, here we are going to start from the trigger and work our way to the SNS topic. So let’s begin!
Adding CloudWatch Trigger
When you create a new project, a skeleton lambda handler is auto generated by Sigma. On that you can see that the event variable is highlighted in red and also a red colored trigger icon appears in front of it, indicating that we haven’t configured any triggers for the lambda yet. So let’s add a CloudWatch trigger.
As shown above, first click on the CloudWatch resource on the left panel, then drag and drop it on top of the event variable to get the configuration panel for that. On the configuration panel, first let’s provide a name for the rule. After that click Schedule tab under the rule type and then configure 1 hour as our rate expression. Alternatively you can define a cron expression as well. Finally click on Inject button to add the trigger and you will see that the trigger icon has now turned into green.
Getting a list of files in S3 bucket
Our next objective is to define a new S3 bucket and implement the functionality to list the currently existing files of that.
For that, first drag and drop a S3 resource from left panel to the necessary line of the editor, where you want to add the code block. Then on the configuration panel, switch to New Bucket tab and first provide a name for the new bucket. Then select List Objects from the operation drop-down, and optionally provide a values maximum number of results and a file name prefix to filter results from. Once you click Inject button after the configurations, you can see that Sigma has auto generated the S3 list object code for you. All you have to do is to define the application specific logic inside the callback functions of the generated code.
After with that application specific logic added, our current code looks like this.
Note that I have also defined a skeleton function named sendNotification() to publish SNS notifications, which is yet to be implemented as well.
Deleting the processed file from S3 bucket
Now we have added the code for listing and processing the files of the S3 bucket. So our next task is to implement the functionality to delete the processed files. In the above example code, it should be done at line number 37.
Since we are going to use the same S3 bucket defined in the previous step in this operation as well, it gets much easier. If you expand the S3 resources list from the left panel, you can see an entry with the defined bucket name. So drag and drop it to line 37 and you will see that the bucket name is listed under the existing buckets and it has been automatically selected as well.
Then select Delete Object from the operation drop-down and then we need to provide the name of the file to be deleted. If you refer the code snippet above, you can see that at this point, we already have this file name in a variable called fileName. To assign that varriable as the value of this field, we need to indicate to Sigma that we are going to use a variable.
For that, first type @{ characters to the field and Sigma will show a list of variables available at this point. Then we can select fileName variable from that list and Sigma will add the closing brace } for you. After that click Inject button to generate the code.
Do you know?
Some fields in Sigma configuration panels support defining a variable as their value. These fields can be identified by the (x) icon at the right end of the field. In such a field, you can define the variable in the format
@{your-var}.
After the code is generated, we can implement the callback functions as per our requirement. The full code at this point will look like below.
Adding SNS notifications
Now from the lambda implementation point of view, only thing left to do is to define a SNS topic, add an email as a subscription to that and use it to publish notifications.
To implement this, first drag and drop a SNS resource from left panel into the sendNotification() function. Then go to the New Topic tab and define a new topic by providing a topic name and a display name. After that click on the Subscriptions button and add your email address as an email subscription endpoint to this topic.
As the operation, select Publish Message and specifiy the subject and message as variables. Finally click Inject button to generate the code. As we have done earlier, then we can implement the callback functions as we need. At the end, the sendNotification() function should look like below.
The complete code will look as below.
Deploying the Solution
Since we have completed developing our solution, now we need to deploy it. For that click on the Deploy Project button at the tool bar.
If you already haven’t done so, this will first ask you to integrate your GitHub account with Sigma and then the code will be committed to a repository under that account. After that, the Lambda deployment artifact will be built using AWS CodeBuild and finally the deployment process will start.
Sigma will first evaluate which deployment changes are required by your project and list them as follows.
Here you can see that the Lambda Function, the CloudWatch rule, the S3 bucket and the SNS topic are going to be created. And Sigma will automatically derive the necessary execution permissions for the Lambda based on the code we developed and add an IAM role with those permissions as well.
Once you click on Execute button, these changes will be deployed to your AWS account and our batch file processing solution will be up and running within few minutes.
Call To Action
- Clap. Appreciate and let others find this article.
- Comment. Share your views on this article.
- Follow me. Udith Gunaratna to receive updates on articles like this.
- Keep in touch. LinkedIn, Twitter
