Deploy AWS Lambda (Node.js) with Serverless and read data from S3

Finally this is my first contribution here for the Medium community. Because I have gained really a lot of knowledge and practical experience from other people on this platform I want to be a part of it and give something back. And finally say thank you!

What are we going to do?

With this article I want to share my experience with AWS Lambda in combination with Serverless Framwork and AWS S3. At the end we have a Lambda function reading csv data located in a S3 bucket, convert them into JSON format and provide it via a public REST endpoint. It does not raise the claim to be the best solution rather than show you what my experience was and how I implemented it on my first try.

Step by step we are going to do the following:

1. Create S3 Bucket and upload csv file
2. Install Serverless Framework locally on your machine
3. Create and configure the Serverless project (Node.js/Express.js)
4. Add role and permissions
5. Deploy Lambda on AWS

I assume you already have some experience with AWS and you know the basics like creating a role. If not please have a look at the official AWS documentation.

Architecture

First let’s have a look at the overall architecture of what we are going to build.

Architecture

As you can see the architecture is actually really simple. The client sends a request to an endpoint provided by the AWS API gateway which forwards it to our Lambda function. Let’s start!

1. Create S3 Bucket

In this step we are going to create a S3 bucket which contains the .csv file from where we want to read the data. Because I assume you are a little bit familiar with AWS I think you know how to create a S3 bucket. You can do it either with the AWS console or via cli.

Name the bucket like you want (or can because it has to be unique as you know). I named my bucket ‘MediumBucketLambda’.

Ok now upload a .csv file to this bucket. I just downloaded a sample file here and renamed it to ‘salesrecords.csv’.
Don’t use the biggest one (100 records is enough). The bucket/file doesn’t have to be public available.

Note: If you want to you use your own .csv file and you just copy and paste the code later here ensure you have a .csv file separating the data with a comma not a semicolon.

2. Install Serverless Framework

Because we don’ t want to set up and configure the Lambda and API gateway manually and especially not separately we use the Serverless Framework which is doing all the work for us. Serverless Framework is an open source framework (Serverless Framework Pro costs a monthly fee). What it does it creates a CloudFormation Stack and deploys everything we need to make it run. We just need to specify a single .yaml file.

First of all we need to install the latest version via npm:

$ sudo npm i serverless -g

And now run:

$ serverless version 

Note: Instead of the command ‘severless’ you can also use the shorter command ‘sls’

You should see something like this:

Serverles version

Ok! Now you are ready to use the serverless cli!

Serverless needs to know to what AWS account you want to deploy your Lambda. We don’t use AWS cli profiles instead we set the IAM user of the AWS account where we want to deploy our Lambda to the serverless config credentials:

$ sls config credentials --provider aws --key KEY --secret SECRET

As you probably know you get the key and secret from the AWS IAM console.
Now every time you deploy something with Serverless this account and IAM user is going to be used.

Note: Please make sure your IAM user has the proper permissions to deploy the Lambda!

Of course if you would work with different AWS accounts and different users and roles where you need to deploy your Lambdas you would managed everything with AWS CLI profiles/credentials (/.aws/credentials and /.aws/config) rather than setting one default account.

3. Create and configure serverless project (Node.js/Express.js)

After we setup the Serverless cli let’s produce some code. We want to create a function which reads data from a S3 bucket in the same account where the Lambda is deployed.

Frist create the folder for your project locally:

$ mkdir serverless_project

Switch into the severless_project directory and run

$ sls create --template aws-nodejs

This command creates two files:

• handler.js
• serverless.yaml

First we delete everything what’s in the handler.js and replace it with the following code.

handler.js

What are we doing here?
We just creating an Express application and a GET route where we read our file from S3, convert it into JSON and provide it to the client.

Don’t forget to install the require modules and dependencies for you node app via npm:

• aws-sdk
• csvtojson
• express
• serverless-http
• serverless-iam-roles-per-function

Note:
Change the ‘params’ object attributes (Bucket, Key) to your needs. The ‘Bucket’ attribute is just the name of your bucket you have created and the ‘Key’ is name of the .csv file.

As you can see the handler.js contains the code of our Lambda function and the serverless.yaml is the main config file for our Lambda deployment. You can configure a lot of things there but don’t worry we just need three sections for our implementation:

Service:

name service

I will name my service “get-sales-data”. If you want to rename it feel free!

Provider:

At the provider section we keep the values for name (aws) and runtime (nodejs10.x).
Our serverless.yaml at this section look like this now:

Configure provider

We will now add some additional properties at the provider section.
The first one is ‘stage’. We assume we want to deploy theLambda at ‘dev’ stage.

stage:dev

The next property is ‘region’. I am going to deploy it in the region ‘eu-west-1’ (Dublin).

region:eu-west-1

Because I want to know what CloudFormation Stack belongs to my Lambda add the property ‘stackName’ with the value ‘my-lambda-${self:provider.stage}’.

stackName:myserverless-lambda-${self:provider.stage}

Note:
${self:provider.stage} references to the stage value ‘dev’ we defined before. Accordingly my CloudFormation Stack has the name ‘my-lambda-dev’.

Our serverless.yaml looks like this now:

Configure provider

There is still one property missing here — the AWS Lambda execution role.
We will postpone adding the role here to the chapter ‘Prepare and configure roles, permissions and policies in AWS’ where we create the execution role.

Function:

After we configured the service and provider section in the serverless.yaml we are now setting the properties for the function.

At the end the ‘functions’ section is supposed to look like this.

Configure function

The handler points to our handler.js file which contains the function to be invoked (we exported it as “server” of what we get back from sls(app)).
The trigger for the lambda is a http trigger. Therefore we need an http endpoint. Lambda itself cannot expose an http endpoint so we going to spin up an AWS API gateway which is doing this job for us. It will generate an http endpoint and route incoming requests forward to our lambda. We specified the path attribute with “/” so every time we call the base path of the http endpoint the lambda is going to be invoked. Additionally we enable CORS and set the ‘private’ attribute value to false. If you want to secure your function with an api key set it simply to true. The functions will be invoked with any http method.

4. Add role and permissions

For me on of the most tricky part of AWS (after switching from Azure to AWS) was to get familiar with setting the right roles, permissions and policies to make everything work.

We need a role for our Lambda function which has the permission to access our S3 bucket we have created before. Because this role will be used for this function only the best way to create this role is to do it also with the Serverless Framework.

To do so, we already installed the prerequsite for that: The “serverless-iam-roles-per-function” npm package. What we need to do to use it is to include it in the serverless.yaml under the “plugin” section right after the “service” section.

With this plugin in place we now can add the role and proper permissions directly under the “function” section were we already defined our function.

Add IAM role and permissons

Note: Don’t forget to add your own S3 bucket name.

5. Deploy Lambda on AWS

Now we are ready do deploy the Lambda on AWS!

Run the command:

$ sls deploy

It takes a little bit if everything is build and the CloudFormation is executed (Serverless does nothing else than creating a CloudFormation Stack). If everything works fine you created a Lamba function and an api gateway which exposes your functions. Congrats!
You see the http endpoint either in your Terminal/Bash after the deployment is finished or in the AWS Lambda Console clicking on API gateway.

Call the endpoint with your browser and you should see the .csv data converted to JSON.

To remove everything (Lambda and IAM role) just execute:

$ sls remove

And everything you deployed via Severless (defined within the serverless.yaml) will be destroyed.

You can find the code used within this articel at GitHub.
Feel free to clone/fork it!