Guide to connecting AWS Lambda to S3 with Pulumi

Adam Boaz Becker
Nov 4, 2019 · 11 min read

Infrastructure As Code: Background

In less than 10 minutes, I will show you how you can put together a scalable API endpoint using AWS API Gateway that will write any payload to an S3 bucket without visiting the AWS console even once!

When I started building the infrastructure for Call Time’s Data Engineering, I hadn’t yet realized the importance of writing Infrastructure As Code. Infrastructure As Code allows you to automate your resource deployment using code and not using a web console.

Spinning up and tearing down your environment using code gives you the confidence that two environments look exactly the same or are only as different as you wish them to be. This allows you to run tests in isolated environments without disrupting your infrastructure.

Imagine crafting, by hand, even a modest infrastructure that looks like this:

Courtesy of CloudCraft.co

Now imagine having to recreate it and crossing your fingers that every resource is glued together properly. Not fun!

None of this should be new — scripting your resource deployment is industry standard. At Call Time, we believe that if you touch the AWS Console, you did something wrong.

Infrastructure As Code Tools

Some tools for managing your cloud resources include AWS CloudFormation, Terraform, and Pulumi. Terraform and Pulumi allow you to choose a cloud provider other than AWS. For this exercise, we’ll use the open source platform Pulumi.

Pulumi

There are several benefits to using Pulumi:

  1. You don’t have to learn a specific DSL or manipulate YAML files — you can script using your favorite language: Python, Go, TypeScript, JavaScript
  2. This layer of abstraction allows you to reuse components
  3. You can view your resources online on pulumi.com
  4. Great documentation — start here

Exercise

This exercise will show us how to create an endpoint that will write to S3 whatever will be POSTed to it. Schematically, here is what we’ll be building:

Image curtesy of me using CloudCraft.co

The API Gateway will reveal an HTTP endpoint that when called, will activate an AWS Lambda that writes its payload to an S3 bucket.

Let’s build it using Pulumi — installing Pulumi will take 2 minutes.

Step 1: Set up Pulumi

$ brew install pulumi

We’ll be using Node.JS for this exercise, so make sure you have that installed. You can check your installation using

$ node --version

If you need to download it, visit the installation page.

Step 2: Configure AWS

Pulumi needs the keys to your AWS account. Create an AWS account if you don’t already have one.

The easiest way to connect Pulumi and AWS is to download the AWS CLI (pip install awscli) and then typing aws configure. This creates a default profile for AWS to use in the command line. It’s this default profile that Pulumi will read.

When prompted, type in your credentials and region:

Great! Now Pulumi can read your credentials and deploy your resources.

Step 3: Create your Pulumi project

Get started with Pulumi quickly by creating a new project. Type the following into a directory of your choosing:

$ pulumi new aws-javascript

This command will prompt you to enter some basic information about your project. For reference, a resource is a cloud object (like an EC2 server), a project is a collection of resources that are deployed together (think of it like a GitHub repo), a stack is the name of the stage you wish to deploy (like dev or prod or staging). Stacks are different from one another by the different configurations they have (like only 1 EC2 server for dev and 2 for prod).

Exploring the project

Let’s spend a moment exploring the project.

Index.js — This file is the entry point for the Pulumi project. Whatever you see here will be deployed when you run pulumi run .

Pulumi.dev.yaml — This file is like our .env file. You can type into it our environmental variables and your stack configuration.

Step 4: Run our deployment — Creating our S3 Bucket

Notice how Index.js arrives with a few default lines.

// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.Bucket("my-bucket");

// Export the name of the bucket
exports.bucketName = bucket.id;

These lines simply create a bucket for us, titled “my-bucket”. Let’s see if it actually works. Type pulumi up .

Great, it worked! We now have a bucket in S3. Let’s see it in action.

Yep, it’s here!

Why does it have this weird name though? That’s because we didn’t feed in a name for the bucket, we just named the pulumi resource so pulumi did some guess-work and came up with some name that will be unique.

In order to refer to the resource itself, we need more arguments. In general, resources are defined as follows:

let res = new Resource(name, args, options)

To find out which args the Bucket resource can take, we have to visit the docs.

For S3, we can feed it a Bucket argument.

// Index.jsconst STACK = pulumi.getStack();

// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.Bucket("payloads-bucket", {
bucket: `api-meetup-bucket-${STACK}`
});

Notice how we also added the name of the STACK so that we have a clear separation of stacks. Let’s run pulumi destroy to remove that old bucket and instead, insert the code above.

Great, now let’s Pulumi up again. Search for it in the console.

Nice. The bucket looks great! Now we’ll create the lambda function that will write to this bucket.

Step 5: Create a Lambda function

Creating the lambda resource is fairly straight-forward as well. It looks like this:

// Index.jsconst lambda = new aws.lambda.CallbackFunction(`payloads-api-meetup-lambda`, {
name: `payloads-api-meetup-lambda-${STACK}`,
runtime: "nodejs8.10",
role: lambdaRole,
callback: lambdaFunction
})

We simply specified the runtime (nodejs8.10), we named the resource in the same way we named the bucket resource, and we assigned a lambda function name using the STACK.

How did we know which options to feed in? Again, we can just find them in the Pulumi Docs.

There are two interesting parameters to look at, role and callback.

Role

We need to create a role for the lambda so that it has permissions to interact with other resources.

Before we know which policies to assign to that role, let’s just create an empty role.

// Index.jsconst lambdaRole = new aws.iam.Role(`role-payloads-api`, {
assumeRolePolicy: `{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow"
}
]
}
`,
})

Notice how we just created the role using the new aws.iam.Role() and “assumed” a fairly empty Policy. That was easy! Now let’s look at the callback argument.

Callback

The resource we decided to use is called new aws.lambda.CallbackFunction . That’s an interesting way to create a lambda function. It means that we can define, in Javascript, a function and then feed it into lambda.

Some limitations of this are obvious: if you have a fairly involved application code you might want to zip it first, and upload it to S3, and use a different method to create lambda.

Our Callback function is fairly straightforward, but let’s build up to it, starting with the wrapper for this function.

// Lambda function
const lambdaFunction = async (event) => {
return {
statusCode: 200,
body: "Success"
}
}

This is very raw — it simply accepts an event argument and returns an HTTP success object.

Let’s add some flesh to it a bit more:

// Lambda functionconst lambdaFunction = async (event) => {
// decode the body of the event
const payloadBuffer = new Buffer(event.body, 'base64')
const payload = payloadBuffer.toString('ascii')

return {
statusCode: 200,
body: "Success"
}
}

Now we can read the payload that was fed into the lambda. We’re getting there!

How about writing it into the S3 bucket?

To write to S3, we should use the aws-sdk in Node. The structure of such an argument is this (using a Promise):

await new Promise((resolve, reject) => {
s3.putObject(putParams, function (err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
})

The putParams in our case is this:

const putParams = {
Bucket: process.env.S3_BUCKET, // We'll read the .env variable
Key: `${new Date().getTime()}.json`, // Our key is a timestamp
Body: payload
}

In order to use s3 we need to import the aws-sdk library.

const AWS = require('aws-sdk')
const s3 = new AWS.S3()

Altogether, our function looks like this:

// Lambda functionconst lambdaFunction = async (event) => {
const AWS = require('aws-sdk')
const s3 = new AWS.S3()
// decode the body of the event
const payloadBuffer = new Buffer(event.body, 'base64')
const payload = payloadBuffer.toString('ascii')
const putParams = {
Bucket: process.env.S3_BUCKET, // We'll read the .env variable
Key: `${new Date().getTime()}.json`, // We'll use the timestamp
Body: payload
}

await new Promise((resolve, reject) => {
s3.putObject(putParams, function (err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
})
return {
statusCode: 200,
body: "Success"
}
}

Our lambda function is now finished!

But notice the line that says process.env.BUCKET_NAME . That line expects our function to have access to an environmental variable called BUCKET_NAME. To expose that variable, we have to feed it into the lambda at creation time:

Let’s update Index.js

// Index.jsconst lambda = new aws.lambda.CallbackFunction(`payloads-api-meetup-lambda`, {
name: `payloads-api-meetup-lambda-${STACK}`,
runtime: "nodejs8.10",
role: lambdaRole,
callback: lambdaFunction,
environment: {
variables: {
S3_BUCKET: bucket.id // Look how bucket refers to the bucket we just created - nice!
}
},
})

Type pulumi up and look at the console.

Nice! It’s up there. But this lambda is a bit isolated — we have no way of triggering it. Let’s hook it up to an API Gateway so we can get a URL.

Step 6: Build the API Gateway

// Index.js// Create our API
let apiGateway = new awsx.apigateway.API(`payloads-api-meetup-api-gateway`, {
routes: [
{
path: "/post_to_s3",
method: "POST",
eventHandler: lambda
}
]
})
// Let's spit out what the URL is:
exports.apiGateway = apiGateway.url

Type pulumi up and you should see your URL!

pulumi up...[Lots of stuff]...Outputs:
+ apiGateway: "https://h96a00as2asdasd3m83c.execute-api.us-west-2.amazonaws.com/stage/"

Nice! If we visit the console again, we’ll see that we can trigger the lambda using the API Gateway. Look!

Cool!

Notice how lambda can be triggered by the API Gateway, but it can’t interact with S3. Why not?

The reason is that we haven’t yet written a permission for the lambda role to write to S3. Let’s add that now.

Step 7: Allow the lambda role to interact with our bucket

We first have to construct the policy for interacting with S3.

// Policy for allowing Lambda to interact with S3
const lambdaS3Policy = new aws.iam.Policy(`post-to-s3-policy`, {
description: "IAM policy for Lambda to interact with S3",
path: "/",
policy: bucket.arn.apply(bucketArn => `{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:PutObject", // Very restrictive policy
"Resource": "${bucketArn}/*",
"Effect": "Allow"
}
]}`)
})

Now that we created the policy, attach it to the Lambda Role.

// Attach the policies to the Lambda role
new aws.iam.RolePolicyAttachment(`post-to-s3-policy-attachment`, {
policyArn: lambdaS3Policy.arn,
role: lambdaRole.name
})

Should be enough! Let’s run pulumi up and view it in the Console.

Step 8: Test it out!

Let’s POST a request in Postman and check whether it made it into S3.

Hit “Send” and visit your S3 bucket.

Step 9: Tear it down

You can just as easily destroy your stack. Simply type pulumi destroy and the entire stack will be torn down.

Next steps

You’ve now completed the entire setup — good job! There are many things you might want to consider as next steps:

  • Separate your rather large file into lots of smaller components — perhaps one file for all the S3 buckets, one for the permissions, etc. — or even nested into folders for the separate functions that each cluster of resources serves
  • You might want to use HTTPS instead of HTTP
  • You will want to grant your lambda the permission to write logs into Cloudwatch so you can debug it more easily
  • Learn how to set up different Pulumi projects and build inter-project dependencies
  • Commit your project to GitHub

Here is theIndex.js file in its entirety:

"use strict";
const pulumi = require("@pulumi/pulumi");
const aws = require("@pulumi/aws");
const awsx = require("@pulumi/awsx");

const STACK = pulumi.getStack();

// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.Bucket("payloads-bucket", {
bucket: `api-meetup-bucket-${STACK}`
});


const lambdaRole = new aws.iam.Role(`role-payloads-api`, {
assumeRolePolicy: `{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow"
}
]
}
`,
})

// Policy for allowing Lambda to interact with S3
const lambdaS3Policy = new aws.iam.Policy(`post-to-s3-policy`, {
description: "IAM policy for Lambda to interact with S3",
path: "/",
policy: bucket.arn.apply(bucketArn => `{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:PutObject",
"Resource": "${bucketArn}/*",
"Effect": "Allow"
}
]}`)
})

// Attach the policies to the Lambda role
new aws.iam.RolePolicyAttachment(`post-to-s3-policy-attachment`, {
policyArn: lambdaS3Policy.arn,
role: lambdaRole.name
})


const lambdaFunction = async (event) => {
const AWS = require('aws-sdk')
const s3 = new AWS.S3()
// decode the body of the event
const payloadBuffer = new Buffer(event.body, 'base64')
const payload = payloadBuffer.toString('ascii')
const putParams = {
Bucket: process.env.S3_BUCKET, // We'll read the .env variable
Key: `${new Date().getTime()}.json`, // We'll use the timestamp
Body: payload
}

await new Promise((resolve, reject) => {
s3.putObject(putParams, function (err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
})
return {
statusCode: 200,
body: "Success"
}
}
const lambda = new aws.lambda.CallbackFunction(`payloads-api-meetup-lambda`, {
name: `payloads-api-meetup-lambda-${STACK}`,
runtime: "nodejs8.10",
role: lambdaRole,
callback: lambdaFunction,
environment: {
variables: {
S3_BUCKET: bucket.id
}
},
})

// create API
let apiGateway = new awsx.apigateway.API(`payloads-api-meetup-api-gateway`, {
routes: [
{
path: "/post_to_s3",
method: "POST",
eventHandler: lambda
}
]
})


// Export the name of the bucket
exports.bucketName = bucket.id
exports.apiGateway = apiGateway.url

If you have any questions, I’d love to be helpful! Find me at adamboazbecker@gmail.com. Good luck!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade