AWS Lambda Image Downloader With S3 Storage

Introduction

In this article we will create an AWS Lambda function which will fetch image from url passed via API endpoint and after some file checks the image will be saved to AWS S3 storage. Code will be written in JavaScript for Node.js environment and Serverless framework will help us with deployment to AWS.

Serverless ~ Lambda ~ S3

Prerequisites

• created AWS account
• installed Node.js and NPM on your local machine

Flow diagram of Lambda function

Solution

If you want just the code without reading this article you can download it here: https://github.com/Emetrop/lambda-image-downloader

Creating S3 bucket

For storing fetched images we will use cheap Amazon S3 (Simple Storage Service) which can store any kind of data.

Go to AWS console and search for `S3`.

S3 organizes all files into so called `buckets` so let’s create one.

Enter name of your bucket (unique name across all existing buckets on AWS) and select region which is closest to your customers (prefer regions with `-1` postfix).

On the same page below you can allow/block public access to your bucket. Permissions in AWS are a huge topic which we won’t cover in this article so just carry on with the default choice of `Block all public access` and finish creating your bucket by clicking on the button in the bottom.

Now you should see your new bucket in the table below.

Serverless framework

For simple set-up and deployment of our Lambda we will use Serverless framework. Serverless eliminates the need for manual use of AWS console and at the same time we will have all configuration saved in a config file which can be easily reused between other Lambda functions.

Creating new IAM user

Serverless library needs permission to access our AWS account. For this purpose we need to create so-called `IAM` (Identity and Access Management) user.

Go to your AWS console and search for `IAM`.

From left menu select `Users`

On the Users page click the `Add user` button.

Name user, select `Programmatic access` option and click onto `Next: Permission` button.

On another page select `Attach existing policies directly` option, search for `AdministratorAccess` and check found policy. Click on the `Next: Tags` button.

Just confirm the next step.

And confirm creation of the new user.

Note (safely!) somewhere your `Access key ID` and `Secret access key`. It’s important to save these credentials safely as with granted admin permissions someone else could do literally anything with your AWS account.

Setting up Serverless framework

Firstly we need to install globally Serverless NPM package.

$ npm install -g serverless

Now log in to AWS with Serverless library by noted credentials.

$ serverless config credentials --provider aws --key YOUR_ACCESS_KEY --secret YOUR_SECRET_KEY

Creating Lambda boilerplate

Let’s generate boilerplate for our Lambda by Serverless framework.

$ serverless create --template aws-nodejs --path downloaders

If everything goes well you should have created new `downloaders` folder with the following structure.

downloaders
│   .gitignore
│   handler.js
│   serverless.yml

Where in the handler.js file will be our Lambda code fetching image and serverless.yml is configuration file which will allow us to do all needed settings on AWS.

Lambda layers

Lambda layer is zip archive which contains additional code which can be included and shared between many Lambdas. All code in a layer is excluded from deployed package of Lambda function and thus it keeps Lambda package small.

Diagram of relation between layers and Lambda functions

Usage of Lambda layer

We will use Lambda layer for including three NPM packages to our Lambda:

• node-fetch => Node.js fetch polyfill
• image-type => detecting file type of images
• uuid => creating unique image names in bucket

Creating Lambda layer

Create new `layer_basic/nodejs` folders in root of your project.

$ mkdir layer_basic && mkdir layer_basic/nodejs

Move to nodejs folder.

$ cd layer_basic/nodejs

Init new NPM project.

$ npm init -y

And install packages.

$ npm i node-fetch image-type uuid

So now your current folder structure should look like this:

downloaders
│   .gitignore
│   handler.js
│   serverless.yml
│
└───layer_basic
│   │
│   └───nodejs
│   │   package.json
│   │   package-lock.json
│   │
│   └───node_modules
│   │   …

serverless.yml

Below is commented self-explanatory serverless.yml file.

handler.js

And finally commented code of our Lambda function.

Deploying and testing function

For deploying function to AWS just run in root of your project.

$ serverless deploy

After a while you should get success message like below.

Service Information
service: downloaders
stage: dev
region: eu-west-1
stack: downloaders-dev
resources: 12
api keys:
  None
endpoints:
  POST - https://xxx.execute-api.eu-west-1.amazonaws.com/dev/fetch-store-image
functions:
  fetchStoreImage: downloaders-dev-fetchStoreImage
layers:
  basic: arn:aws:lambda:eu-west-1:yyy:layer:basic:1

You can grab the POST endpoint url and try it in terminal with curl.

$ curl --location --request POST ‘https://xxx.execute-api.eu-west-1.amazonaws.com/dev/fetch-store-image' \
  --header ‘Content-Type: application/json’ \
  --data-raw ‘{“url”: "https://xxxxx.jpg"}'

And you should get response like this.

{“success”:true,”key”:”original/7e6bc531–2d9d-4114-a306–09270e6294ff.jpg”}

Now you can go to your bucket in AWS S3 console and check that the image is really there.

Congratulations!

Complete repository can be downloaded here: https://github.com/Emetrop/lambda-image-downloader

Conclusion

With usage of AWS Lambda function, Lambda layer, Serverless framework and AWS S3 we created publicly accessible POST API endpoint which downloads and stores images to S3 storage.