How to upload files to Amazon s3 using NodeJs, Lambda and API Gateway

Uploading binary to the s3 bucket using Amazon Lambda and API Gateway can be tricky sometimes and I’m going to share with you how I was able to do that.

Working with Accounteer has been a blessing to me in terms of taking up challenges. It has enabled me to explore new technologies and how to make use of a lot of Amazon web services.

I had the challenge to create an endpoint that will allow Accounteer users to upload files to their new or existing invoices, offers and purchases on the main app via the API.

I started my research and went through the Amazon documentation and it was stated that users can upload binaries directly to s3 using Lambda but to my amazement, it’s difficult to do that because Amazon API gateway passthrough only allows JSON as Content-Type.

Let me start by showing you how I did it. If you will like to follow my steps you must be a registered user on Amazon and must have created a bucket on your s3.

After you have created your bucket, there are other few steps that will be needed and I’m going to show you.

Firstly, you will need to open your bucket, click on the properties then edit the policy and permissions. This will allow the user to read and write to the bucket.

Bucket Policy

{
 “Sid”: “PublicReadGetObject”,
 “Effect”: “Allow”,
 “Principal”: “*”,
 “Action”: “s3:GetObject”,
 “Resource”: “arn:aws:s3:::bucket-name/*”
 }

Bucket permission

<CORSConfiguration xmlns=”http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
 <AllowedOrigin>*</AllowedOrigin>
 <AllowedMethod>PUT</AllowedMethod>
 <AllowedMethod>POST</AllowedMethod>
 <AllowedMethod>GET</AllowedMethod>
 <AllowedMethod>HEAD</AllowedMethod>
 <MaxAgeSeconds>3000</MaxAgeSeconds>
 <AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
</CORSConfiguration>

After setting the above configuration on your new or existing bucket, the next thing is to create our Node app that Lambda can use to handle the file upload.

One of the easiest ways I used to upload files to S3 using Lambda is to convert it to a base64 encoded string and pass it to the buffer and then to the s3 putObject method and it’s as simple as that.

Dependencies

  • aws-sdk
  • moment
  • file-type
  • sha1
The actual lambda that handles the file upload
This code gets you the file properties e.g File size, mimeType and so on

After creating the code above you will need to zip it for later use.

The next thing that we need to do is create our Lambda function on Amazon and also create an endpoint that’s expecting a POST request to the Lambda we have created.

Create your lambda and import your code here

After you might have imported your code and given your Lambda a meaningful name then you can to proceed to create the endpoint where you will be able to send a POST request and get your file object uploaded to s3.

You need to browse through Amazon web services and click API gateway.

Create the API here and give it a meaningful name

After creating the API, you can proceed to create an endpoint for the file upload.

Select your region and attach your lambda function

After selecting your region and connecting your lambda function then we proceed to API gateway configuration.

The above allows you to set your request and response

They are 4 parts that complement the API gateway;

Method Request

Method Request allows you to intercept the request from the user, here you can set the Authorization as well as your own custom Authorizer in case you want to use JWT for token authorization. You can as well set the headers you want.

Integration Request

This where the real body passthrough takes place. You will need to configure the body mapping template. This will enable the API gateway to recognize your request to the lambda event.

Set your body mapping template here

Method Response

This is where you need to set your response status code e.g

  • 404
  • 500
  • 200 — default
  • 401
Set your status codes here

Integration Request

This phase allows you to set your mapping to respond to the status codes you set in the Method Response. You will need to construct a regex for each status code.

Respond to the 404 code and form the mapping template to format the error message

After completing the above steps, the next is to deploy our API and use POSTMAN to consume the endpoint

Version your API here
a sample POST request to the lambda function

You can use cloud watch to monitor the response in case there is an error otherwise you should get the file URL back

{ “file_url”: “bucketpath+fullfilepath”}

Thank you very much for going through the tutorial. Please feel free to drop comments and errata.