AWS Lambda Image Downloader With S3 Storage

Martin Babinec
May 17, 2020 · 5 min read

Introduction

In this article we will create an AWS Lambda function which will fetch image from url passed via API endpoint and after some file checks the image will be saved to AWS S3 storage. Code will be written in JavaScript for Node.js environment and Serverless framework will help us with deployment to AWS.

Image for post
Image for post
Serverless ~ Lambda ~ S3

Prerequisites

  • created AWS account

Flow diagram of Lambda function

Image for post
Image for post

Solution

If you want just the code without reading this article you can download it here: https://github.com/Emetrop/lambda-image-downloader

Creating S3 bucket

For storing fetched images we will use cheap Amazon S3 (Simple Storage Service) which can store any kind of data.

Go to AWS console and search for `S3`.

Image for post
Image for post

S3 organizes all files into so called `buckets` so let’s create one.

Image for post
Image for post

Enter name of your bucket (unique name across all existing buckets on AWS) and select region which is closest to your customers (prefer regions with `-1` postfix).

Image for post
Image for post

On the same page below you can allow/block public access to your bucket. Permissions in AWS are a huge topic which we won’t cover in this article so just carry on with the default choice of `Block all public access` and finish creating your bucket by clicking on the button in the bottom.

Image for post
Image for post

Now you should see your new bucket in the table below.

Image for post
Image for post

Serverless framework

For simple set-up and deployment of our Lambda we will use Serverless framework. Serverless eliminates the need for manual use of AWS console and at the same time we will have all configuration saved in a config file which can be easily reused between other Lambda functions.

Creating new IAM user

Serverless library needs permission to access our AWS account. For this purpose we need to create so-called `IAM` (Identity and Access Management) user.

Go to your AWS console and search for `IAM`.

Image for post
Image for post

From left menu select `Users`

Image for post
Image for post

On the Users page click the `Add user` button.

Image for post
Image for post

Name user, select `Programmatic access` option and click onto `Next: Permission` button.

Image for post
Image for post

On another page select `Attach existing policies directly` option, search for `AdministratorAccess` and check found policy. Click on the `Next: Tags` button.

Image for post
Image for post

Just confirm the next step.

Image for post
Image for post

And confirm creation of the new user.

Image for post
Image for post

Note (safely!) somewhere your `Access key ID` and `Secret access key`. It’s important to save these credentials safely as with granted admin permissions someone else could do literally anything with your AWS account.

Image for post
Image for post

Setting up Serverless framework

Firstly we need to install globally Serverless NPM package.

$ npm install -g serverless

Now log in to AWS with Serverless library by noted credentials.

$ serverless config credentials --provider aws --key YOUR_ACCESS_KEY --secret YOUR_SECRET_KEY

Creating Lambda boilerplate

Let’s generate boilerplate for our Lambda by Serverless framework.

$ serverless create --template aws-nodejs --path downloaders

If everything goes well you should have created new `downloaders` folder with the following structure.

downloaders
│ .gitignore
│ handler.js
│ serverless.yml

Where in the handler.js file will be our Lambda code fetching image and serverless.yml is configuration file which will allow us to do all needed settings on AWS.

Lambda layers

Lambda layer is zip archive which contains additional code which can be included and shared between many Lambdas. All code in a layer is excluded from deployed package of Lambda function and thus it keeps Lambda package small.

Image for post
Image for post
Diagram of relation between layers and Lambda functions

Usage of Lambda layer

We will use Lambda layer for including three NPM packages to our Lambda:

  • node-fetch => Node.js fetch polyfill

Creating Lambda layer

Create new `layer_basic/nodejs` folders in root of your project.

$ mkdir layer_basic && mkdir layer_basic/nodejs

Move to nodejs folder.

$ cd layer_basic/nodejs

Init new NPM project.

$ npm init -y

And install packages.

$ npm i node-fetch image-type uuid

So now your current folder structure should look like this:

downloaders
│ .gitignore
│ handler.js
│ serverless.yml

└───layer_basic
│ │
│ └───nodejs
│ │ package.json
│ │ package-lock.json
│ │
│ └───node_modules
│ │ …

serverless.yml

Below is commented self-explanatory serverless.yml file.

handler.js

And finally commented code of our Lambda function.

Deploying and testing function

For deploying function to AWS just run in root of your project.

$ serverless deploy

After a while you should get success message like below.

Service Information
service: downloaders
stage: dev
region: eu-west-1
stack: downloaders-dev
resources: 12
api keys:
None
endpoints:
POST - https://xxx.execute-api.eu-west-1.amazonaws.com/dev/fetch-store-image
functions:
fetchStoreImage: downloaders-dev-fetchStoreImage
layers:
basic: arn:aws:lambda:eu-west-1:yyy:layer:basic:1

You can grab the POST endpoint url and try it in terminal with curl.

$ curl --location --request POST ‘https://xxx.execute-api.eu-west-1.amazonaws.com/dev/fetch-store-image' \
--header ‘Content-Type: application/json’ \
--data-raw ‘{“url”: "https://xxxxx.jpg"}'

And you should get response like this.

{“success”:true,”key”:”original/7e6bc531–2d9d-4114-a306–09270e6294ff.jpg”}

Now you can go to your bucket in AWS S3 console and check that the image is really there.

Image for post
Image for post

Congratulations!

Image for post
Image for post

Complete repository can be downloaded here: https://github.com/Emetrop/lambda-image-downloader

Conclusion

With usage of AWS Lambda function, Lambda layer, Serverless framework and AWS S3 we created publicly accessible POST API endpoint which downloads and stores images to S3 storage.

The Startup

Medium's largest active publication, followed by +756K people. Follow to join our community.

Martin Babinec

Written by

An eager software developer daily going beyond event horizon and back. @emetrop

The Startup

Medium's largest active publication, followed by +756K people. Follow to join our community.

Martin Babinec

Written by

An eager software developer daily going beyond event horizon and back. @emetrop

The Startup

Medium's largest active publication, followed by +756K people. Follow to join our community.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store