AWS Lambda: A New Generation of Cloud Compute Services

We have been building an internal tool to manage and troubleshoot our product logistics pipeline. We embarked on this task with a small team and a limited amount of time to become feature complete. While building it we wanted to specifically consider automating many repetitive administrative tasks/chores. Our goal was to develop a product that was not only effective, but also low-cost.

This is where AWS Lambda shines. AWS Lambda is an event-driven compute service by Amazon; it can deploy and run your code in response to events. No provisioning or deployments are required. I like the idea of an event-driven programming model as it emphasizes decoupling between the sender and receiver systems/modules, eventually leading to a microservices driven architecture. I created a processing pipeline for genotype data at Athletigen Inc. to take this service for a test drive.

Use Cases

Allow me to motivate the discussion with some practical use cases for AWS Lambda:

• Crawling and indexing internet: A crawler can generate results; store them in S3 which can then be configured to trigger a Lambda handler function to process the new item; add the new links in the queue; and finally - clean up after.
• Scheduled tasks: Set up scheduled triggers in CloudWatch, which works as a cron job in your AWS infrastructure and sets up a Lambda function to handle those events to perform some calculation on regular basis. For example, you could perform a count on a database table and update somewhere else for analytics purposes. [Example using serverless]
• Processing Kinesis streams: You can use Lambda as post-processing system for data coming from a Kinesis stream. Log cleaning and processing is an ideal fit here.
• Media/File transformations: On arrival of a new file in s3 bucket, Lambda can be used to transform or compress images (and other media/files).
• Web App: Lambda can be used in combination with API Gateway to replace traditional backend infrastructure. [Example chat application build using Lambda and API Gateway]

If you’re planning on using Lambda for event handling within the AWS world, your life will be easier. Although there are other services, like Webtasks, which will not constraint you to Amazon as your Cloud provider.

Our current use of Lambda involves transferring data between two services on a daily basis. Our DNA kit sales are managed by an external shop service, so we needed a way to automatically bring in the external orders to our internal system and alert the fulfillment lab (external) via email. Using AWS Lambda and SES made this solution a breeze to maintain.

Once you have a function ready, I suggest using something like Apex Serverless to deploy your Lambda function with almost no effort. Apex is a great tool written to make Lambda deployments and management more usable than what AWS provides out of the box. You can do dry runs, rollback etc. A blog post from the creator of Apex is a great read to get the background and introduction to the features of Apex.

After having deployed a couple of my own Lambda functions and there are some pros and cons of using the service which I want to highlight.

Pros:

• Very cheap compute engine
• Super quick to get your code running in the Cloud
• Effortless monitoring with 1 minute resolution
• Changes to migrate existing automation code are minimal

Cons:

• In Lambda, the runtime choices are limited
• There’s one performance-related knob to tweak, and that’s memory
• Requires developer to bundle libraries and upload the .zip to S3
• There is only one Lambda execution AMI
• Five minute execution limit can be a restriction for workloads like ETL or video processing but should be manageable for most of the event-driven workloads.

Costs:

Lambda has an indefinitely free tier even after your initial free 12 months tier from Amazon which includes 1 Million free requests and 400,000 GB-seconds of compute time per month.

If we compare the cheapest options in EC2 and Lambda to run a 3 min job on a daily basis, EC2 will cost approximately 6.28$/Month whereas Lambda costs 0.02$/month. This is because Lambda pricing is per execution request and EC2 pricing based on % utilization and uptime of the instance. To read more detailed calculation refer to [1], [2], [3].

Here are some useful links related to this server less ecosystem and AWS Lambda.

• Introduction to apex serverless toolkit
• Detailed Cost comparison of Lambda and EC2
• Limitations of AWS Lambda
• Building a web app using serverless architecture
• AWS Lambda used to load data into Redshift Data-warehouse
• Interview with Andrew Templeton who uses AWS Lambda for building production quality web apps

PS: Thanks to Kate and Martin for reviewing and helping me with this article.