Upload binary files to S3 using AWS API Gateway with AWS Lambda

Omer Hanetz
May 12, 2019 · 8 min read

How to use AWS API Gateway endpoint with Python AWS Lambda backend to allow uploads of binary files to your cloud environment.

Table of Contents

Introduction

AWS API Gateway binary support makes it possible to send requests with any content type. In order to make that possible the data is encoded in base64, so some manipulation is required before the data can be handled or stored for future use.

The ability to create an AWS Lambda function as an internal handler makes this manipulation and the integration of the requests with our cloud environment resources much easier.

Requirements

The solution is using an AWS Lambda function written in Python, but a similar solution is possible for other languages supported by AWS Lambda.

Writing the AWS Lambda function

Creating the function

In the Lambda page click on Create function. Choose Author from scratch, type a name, and select Python 3.6 or Python 3.7 runtime. Expand the Permissions section, and choose Create a new role with basic Lambda permissions. This will setup CloudWatch Logs for your Lambda function, so you’ll be able to track executions. However, we also need S3 permissions to write our files. We will handle that later. Click on Create function to finish.

Writing the function code

The Python Lambda code editor comes with a default lambda_function.py file with a lambda_handler method. We will use this default file and method for our example.

In the code editor, delete the content of the lambda_function.py file, and type the following code instead (Don’t forget to replace the placeholders with your S3 bucket name and file path):

import json
import base64
import boto3
BUCKET_NAME = 'YOUR_S3_BUCKET_NAME_HERE'def lambda_handler(event, context):
file_content = base64.b64decode(event['content'])
file_path = 'YOUR_FILE_PATH_HERE'
s3 = boto3.client('s3')
try:
s3_response = s3.put_object(Bucket=BUCKET_NAME, Key=file_path, Body=file_content)
except Exception as e:
raise IOError(e)
return {
'statusCode': 200,
'body': {
'file_path': file_path
}
}

This function receives base64 encoded binary content, and uploads it to a certain bucket in S3. Then it returns the file path on a successful response.

Granting S3 access to the function

In the function management page, go to the Execution role section, and click on View the … role on the IAM console.

In the role IAM page click on Attach policies. Select the AmazonS3FullAccess policy and click on Attach policy.

You can also create a new policy that grants less access, such as write access only, or access to a specific bucket. However, since the access is hard-coded in your Lambda function (and only writes new files), it is safe to use the full access policy here.

Testing the function

In the Lambda function management page click on Test, then select Create new test event, type a name, and replace the sample data with a simple JSON object that has a key named content as follows (The content should be base64 encoded):

{
"content": "c2FtcGxlIHRleHQ="
}

Click on Create to create the test event, and then in the function management page click on Test again to test your function. A successful test should give you a response similar to this:

You should also be able to see your sample file in S3.

Setting up the API Gateway

Creating the API Gateway endpoint

Click on Create API, choose REST, select New API and type a name. Click on Create API to finish.

You will be redirected to the resources page. Here you can define your endpoint paths (defined here as resources), and the methods behavior for each path.

Setting up a POST method endpoint

Then, open the Actions menu again, and this time choose Create Method. In the opened dropdown select POST, and click on the small V icon to confirm.

In the opened form, choose Lambda Function as the integration type, and select your newly created Lambda function by typing its name or ARN identifier. Click on Save, and confirm that you allow API Gateway to invoke your Lambda function.

You will then be redirected to the Method Execution configuration page of your newly created method, where you can handle the flow of the request to the gateway and from the gateway to your environment (in this case, Lambda function), and the response from your environment to the gateway and back to the requester.

You can also test your method execution here, but since we handle binary content, this is not really feasible.

Accepting binary data of certain types

Open your method Integration Request, and expand the Mapping Templates section.

In Request body passthrough choose When there are no templates defined.

Then, for each of your media types click on Add mapping template. In the Content-Type box type your media type identifier (for example, for PDF type application/pdf), and click on the small V icon.

In the template box type the following, then click Save:

{
"content": "$input.body"
}

This will tell your endpoint to pass the body of your request (the binary content) to the Lambda function inside the content property. The content will be passed encoded as base64. Recall that in our Lambda function we got the file content by accessing the event content property and decoding it as base64.

Next we need to add binary media types support to our endpoint. This will determine which media types will be treated as binary (and encoded as base64 to be processed in our Lambda function).

In the endpoint left menu click on Settings, then in the Binary Media Types section, for each of your content types click on Add Binary Media Type and type your content identifier. Click on Save Changes to finish.

Deploying the endpoint

Select [New Stage] as the deployment stage, type a stage name (e.g. v1), and click Deploy.

You will be redirected to the Stages page, and at the top you will see your endpoint Invoke URL.

It’s important to note that while your endpoint is managed by AWS, it is not secured out-of-the-box. Once you have deployed your endpoint it becomes publicly accessible, and any request to the invocation URL with the correct method and data will invoke your Lambda function. I am planning to cover endpoint security in my next post.

Testing the API Gateway

For a cURL command, run something like the following example:

curl --request POST -H "Content-Type: application/pdf" --data-binary "@/path/to/your/file.pdf" https://YOUR_API_GATEWAY_ID.execute-api.us-east-1.amazonaws.com/v1/upload

A successful execution should return the response sent by the Lambda function:

{
"statusCode": 200,
"body": {
"file_path": "sample.txt"
}
}

Summary

This article walked through an implementation of basic AWS API Gateway Endpoint with Python AWS Lambda for a specific use-case of binary file upload. Naturally, there are plenty of other use-cases, there is also much more to add in order to make this implementation production-ready, such as custom domain names, certificates, and authentication (which I am planning to cover next). However, I hope this solution could be a great way to get familiar with AWS API Gateway, and a great starting point.

The Startup

Get smarter at building your thing. Join The Startup’s +788K followers.

Sign up for Top 10 Stories

By The Startup

Get smarter at building your thing. Subscribe to receive The Startup's top 10 most read stories — delivered straight into your inbox, once a week. Take a look.

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Omer Hanetz

Written by

I am writing code ever since I learned how to use a keyboard and I still think that nothing beats the feeling of seeing your code runs as you expect. hanetz.net

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +788K followers.

Omer Hanetz

Written by

I am writing code ever since I learned how to use a keyboard and I still think that nothing beats the feeling of seeing your code runs as you expect. hanetz.net

The Startup

Get smarter at building your thing. Follow to join The Startup’s +8 million monthly readers & +788K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store