I recently came across a use case where I had to add text and resize multiple images a day and convert them to WebP format from JPEG so that they can be served more efficiently on the web.
As an initial approach, I used a simple Node server (in an EC2 box) which used sharp library (more on this later) to process images. Since the use case was more along the lines of processing a maximum of five to ten images at a time, this worked fine.
As the system started to scale up, we had cases where there were requirements to process over a hundred images at once. This was when our application became sluggish and there was an additional overhead of uploading these images to S3 (AWS storage) programmatically.
This was when we explored AWS lambda. AWS Lambda is an event-driven, serverless computing platform provided by Amazon as a part of Amazon Web Services.
Lambda, popularly known as the server-less architecture, lets you run server-side code without depending on the set up of an external box like EC2. The container running the code exits as soon as the code execution is completed.
The main advantage here is that you pay only for the computing power used and data transfer charges, unlike EC2 which charges you by the hour, irrespective of whether or not the system is being used.
Lambda also scales up automatically based on the number of requests received without the requirement of an external load balancer.
Lambda functions are triggered by certain events that can originate from any of the following sources.
- Kinesis — Provides streams for data transfer in bulk, though not an ideal candidate, it still can be used to trigger Lambda.
- SQS — Message Queuing Service. You can trigger Lambda by emitting messages through SQS.
- API gateways — Provides a simple endpoint that can trigger the Lambda function via an HTTP request.
- S3 — Creating, updating, or deleting files in a bucket can be used to trigger Lambda.
For a list of other available Lambda triggers and how they work in-depth, check out the official docs.
It is basically up to you to select how you want to trigger your Lambda functions. Furthermore, Lambda comes with configurations for memory allocation in every function and timeout, so that if a function crashes, you won’t be charged indefinitely.
Every Lambda function internally runs in a container. Every time a function executes, it is spinning up a container that does the task, and the underlying environment is a Linux AMI that can run code in a wide variety of languages.
We’ll be using Node.js (version 10) to demonstrate this.
Our Sample Application
Let’s consider a simple Node app that downloads an image from S3, resizes it to a width of 720px, converts it to WebP, and uploads it to S3 as a processed file.
The sample application will use sharp (a library based on libvips) for image processing. We use sharp since it’s under active development unlike other libraries in Node that work in image processing.
Sharp also has the best benchmark scores among its peers, see the performance.
Consider the flowchart below. Our application will go through the following flow once it gets triggered through an API gateway.
We can break the app into three pieces:
- Downloading the image from S3.
- Processing the image using sharp.
- Uploading the processed image.
Step 1: Downloading the image to a directory in Lambda
This is fairly straightforward, we use AWS’ SDK for Node.
We download the file into a temp directory.
Step 2: Processing the image
Sharp takes the input path as a parameter, it can be chained with other operations such as resize. See here for a more detailed list of available options.
Step 3: Uploading the processed image to S3
We will be using the AWS SDK again for this step.
For an in-depth look into how the AWS SDK works and the APIs available in it, check out this documentation.
Setting Up Lambda
We can break this part down into four steps.
- Creating our Lambda function.
- Creating a KMS key for encryption.
- Adding triggers.
- Linking our packaged code to Lambda.
Now that we’ve got our application ready, we can set up Lambda.
Let’s use an API gateway for this purpose. The choice of the trigger is up to you. We’re using an API for this tutorial since it’s easier to understand. While developing the app in production, I used AWS SQS to trigger my Lambda functions.
Let’s start by creating our Lambda function. You can do this by logging into your AWS console and selecting AWS Lambda.
Make sure you remember the region where you’re creating your Lambda function as we’ll be using the same region for KMS (Key management service).
Since we’ll be using Node, I’ve selected Node 10 as the runtime. Remember to select the option to create a new role, initially, this role will have permissions to write stdout logs to CloudWatch.
The next step is to add additional permissions to the role created, primarily KMS as we’ll be using it for encryption and decryption of our secret keys because we are following best practices.
KMS is a service provided by AWS where we can manage our keys that encrypt and decrypt sensitive info such as access key IDs.
To create a key, we’ll use the console again, navigate to the KMS section in the available services and select the option to create a key.
Once the basic info is filled up, you will have to define the key usage, we will be selecting Lambda’s custom role here since the Lambda function needs to access the key.
Now that we’ve created the key, we can encrypt our data which can be accessed later in the Lambda functions as environment variables.
In your Lambda dashboard, under environment variables, check the enable helpers option and search for the generated KMS key in the KMS key to encrypt input box.
You can now start adding all the env variables you need for your app and they can be accessed directly in your function. For our demonstration, we’ll need to encrypt the access key ID, secret key, and region as this info will be used in downloading and uploading images to S3.
The next step is to add triggers for our application, we’ll be using an API gateway for this purpose. In the Lambda dashboard, select the add trigger functionality.
We’ve not added any security to our API gateway for the sake of simplicity. However, you can add security to restrict the API access to specific IAM users. Once the API endpoint is generated, we can use it to trigger our app.
Our app needs to be packaged with all the dependencies on our local machine. Run the following command:
npm install aws-sdk sharp
This will generate a folder called
node_modules, zip it with our code and upload it to a bucket in S3. Provide the link of this file in Lambda as shown in the image below and save the code.
To test the app, let’s upload an image to an S3 bucket that has open read permissions.
Modify the JSON data above to match your S3 data. Once the Lambda function executes, you’ll be able to see the new file in WebP format appear in the destination bucket as specified in the code.
Lambda is an amazing service that lets you scale with zero effort and helps you execute your functions in your choice of languages. It comes with a lot of flexibility and has typically become a choice for operations like file processing.
It can also help in use cases where setting up a server becomes overkill like a website that has a simple form that sends data to a business owner through email. The free tier in Lambda comes with one million free requests, making your code execution free in most cases.
The complete code is available on GitHub.