Guide to connecting AWS Lambda to S3 with Pulumi
Infrastructure As Code: Background
In less than 10 minutes, I will show you how you can put together a scalable API endpoint using AWS API Gateway that will write any payload to an S3 bucket without visiting the AWS console even once!
When I started building the infrastructure for Call Time’s Data Engineering, I hadn’t yet realized the importance of writing Infrastructure As Code. Infrastructure As Code allows you to automate your resource deployment using code and not using a web console.
Spinning up and tearing down your environment using code gives you the confidence that two environments look exactly the same or are only as different as you wish them to be. This allows you to run tests in isolated environments without disrupting your infrastructure.
Imagine crafting, by hand, even a modest infrastructure that looks like this:

Now imagine having to recreate it and crossing your fingers that every resource is glued together properly. Not fun!
None of this should be new — scripting your resource deployment is industry standard. At Call Time, we believe that if you touch the AWS Console, you did something wrong.
Infrastructure As Code Tools
Some tools for managing your cloud resources include AWS CloudFormation, Terraform, and Pulumi. Terraform and Pulumi allow you to choose a cloud provider other than AWS. For this exercise, we’ll use the open source platform Pulumi.
Pulumi
There are several benefits to using Pulumi:
- You don’t have to learn a specific DSL or manipulate YAML files — you can script using your favorite language: Python, Go, TypeScript, JavaScript
- This layer of abstraction allows you to reuse components
- You can view your resources online on pulumi.com
- Great documentation — start here
Exercise
This exercise will show us how to create an endpoint that will write to S3 whatever will be POSTed to it. Schematically, here is what we’ll be building:

The API Gateway will reveal an HTTP endpoint that when called, will activate an AWS Lambda that writes its payload to an S3 bucket.
Let’s build it using Pulumi — installing Pulumi will take 2 minutes.
Step 1: Set up Pulumi
$ brew install pulumiWe’ll be using Node.JS for this exercise, so make sure you have that installed. You can check your installation using
$ node --versionIf you need to download it, visit the installation page.
Step 2: Configure AWS
Pulumi needs the keys to your AWS account. Create an AWS account if you don’t already have one.
The easiest way to connect Pulumi and AWS is to download the AWS CLI (pip install awscli) and then typing aws configure. This creates a default profile for AWS to use in the command line. It’s this default profile that Pulumi will read.
When prompted, type in your credentials and region:

Great! Now Pulumi can read your credentials and deploy your resources.
Step 3: Create your Pulumi project
Get started with Pulumi quickly by creating a new project. Type the following into a directory of your choosing:
$ pulumi new aws-javascriptThis command will prompt you to enter some basic information about your project. For reference, a resource is a cloud object (like an EC2 server), a project is a collection of resources that are deployed together (think of it like a GitHub repo), a stack is the name of the stage you wish to deploy (like dev or prod or staging). Stacks are different from one another by the different configurations they have (like only 1 EC2 server for dev and 2 for prod).

Exploring the project
Let’s spend a moment exploring the project.
Index.js — This file is the entry point for the Pulumi project. Whatever you see here will be deployed when you run pulumi run .

Pulumi.dev.yaml — This file is like our .env file. You can type into it our environmental variables and your stack configuration.

Step 4: Run our deployment — Creating our S3 Bucket
Notice how Index.js arrives with a few default lines.
// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.Bucket("my-bucket");
// Export the name of the bucket
exports.bucketName = bucket.id;These lines simply create a bucket for us, titled “my-bucket”. Let’s see if it actually works. Type pulumi up .

Great, it worked! We now have a bucket in S3. Let’s see it in action.

Yep, it’s here!
Why does it have this weird name though? That’s because we didn’t feed in a name for the bucket, we just named the pulumi resource so pulumi did some guess-work and came up with some name that will be unique.
In order to refer to the resource itself, we need more arguments. In general, resources are defined as follows:
let res = new Resource(name, args, options)To find out which args the Bucket resource can take, we have to visit the docs.
For S3, we can feed it a Bucket argument.
// Index.jsconst STACK = pulumi.getStack();
// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.Bucket("payloads-bucket", {
bucket: `api-meetup-bucket-${STACK}`
});
Notice how we also added the name of the STACK so that we have a clear separation of stacks. Let’s run pulumi destroy to remove that old bucket and instead, insert the code above.
Great, now let’s Pulumi up again. Search for it in the console.

Nice. The bucket looks great! Now we’ll create the lambda function that will write to this bucket.
Step 5: Create a Lambda function
Creating the lambda resource is fairly straight-forward as well. It looks like this:
// Index.jsconst lambda = new aws.lambda.CallbackFunction(`payloads-api-meetup-lambda`, {
name: `payloads-api-meetup-lambda-${STACK}`,
runtime: "nodejs8.10",
role: lambdaRole,
callback: lambdaFunction
})
We simply specified the runtime (nodejs8.10), we named the resource in the same way we named the bucket resource, and we assigned a lambda function name using the STACK.
How did we know which options to feed in? Again, we can just find them in the Pulumi Docs.
There are two interesting parameters to look at, role and callback.
Role
We need to create a role for the lambda so that it has permissions to interact with other resources.
Before we know which policies to assign to that role, let’s just create an empty role.
// Index.jsconst lambdaRole = new aws.iam.Role(`role-payloads-api`, {
assumeRolePolicy: `{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow"
}
]
}
`,
})
Notice how we just created the role using the new aws.iam.Role() and “assumed” a fairly empty Policy. That was easy! Now let’s look at the callback argument.
Callback
The resource we decided to use is called new aws.lambda.CallbackFunction . That’s an interesting way to create a lambda function. It means that we can define, in Javascript, a function and then feed it into lambda.
Some limitations of this are obvious: if you have a fairly involved application code you might want to zip it first, and upload it to S3, and use a different method to create lambda.
Our Callback function is fairly straightforward, but let’s build up to it, starting with the wrapper for this function.
// Lambda function
const lambdaFunction = async (event) => {
return {
statusCode: 200,
body: "Success"
}
}This is very raw — it simply accepts an event argument and returns an HTTP success object.
Let’s add some flesh to it a bit more:
// Lambda functionconst lambdaFunction = async (event) => {
// decode the body of the event
const payloadBuffer = new Buffer(event.body, 'base64')
const payload = payloadBuffer.toString('ascii')
return {
statusCode: 200,
body: "Success"
}
}
Now we can read the payload that was fed into the lambda. We’re getting there!
How about writing it into the S3 bucket?
To write to S3, we should use the aws-sdk in Node. The structure of such an argument is this (using a Promise):
await new Promise((resolve, reject) => {
s3.putObject(putParams, function (err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
})The putParams in our case is this:
const putParams = {
Bucket: process.env.S3_BUCKET, // We'll read the .env variable
Key: `${new Date().getTime()}.json`, // Our key is a timestamp
Body: payload
}In order to use s3 we need to import the aws-sdk library.
const AWS = require('aws-sdk')
const s3 = new AWS.S3()Altogether, our function looks like this:
// Lambda functionconst lambdaFunction = async (event) => {
const AWS = require('aws-sdk')
const s3 = new AWS.S3() // decode the body of the event
const payloadBuffer = new Buffer(event.body, 'base64')
const payload = payloadBuffer.toString('ascii') const putParams = {
Bucket: process.env.S3_BUCKET, // We'll read the .env variable
Key: `${new Date().getTime()}.json`, // We'll use the timestamp
Body: payload
}
await new Promise((resolve, reject) => {
s3.putObject(putParams, function (err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
}) return {
statusCode: 200,
body: "Success"
}
}
Our lambda function is now finished!
But notice the line that says process.env.BUCKET_NAME . That line expects our function to have access to an environmental variable called BUCKET_NAME. To expose that variable, we have to feed it into the lambda at creation time:
Let’s update Index.js
// Index.jsconst lambda = new aws.lambda.CallbackFunction(`payloads-api-meetup-lambda`, {
name: `payloads-api-meetup-lambda-${STACK}`,
runtime: "nodejs8.10",
role: lambdaRole,
callback: lambdaFunction,
environment: {
variables: {
S3_BUCKET: bucket.id // Look how bucket refers to the bucket we just created - nice!
}
},
})
Type pulumi up and look at the console.

Nice! It’s up there. But this lambda is a bit isolated — we have no way of triggering it. Let’s hook it up to an API Gateway so we can get a URL.
Step 6: Build the API Gateway
// Index.js// Create our API
let apiGateway = new awsx.apigateway.API(`payloads-api-meetup-api-gateway`, {
routes: [
{
path: "/post_to_s3",
method: "POST",
eventHandler: lambda
}
]
})
// Let's spit out what the URL is:
exports.apiGateway = apiGateway.url
Type pulumi up and you should see your URL!
pulumi up...[Lots of stuff]...Outputs:
+ apiGateway: "https://h96a00as2asdasd3m83c.execute-api.us-west-2.amazonaws.com/stage/"
Nice! If we visit the console again, we’ll see that we can trigger the lambda using the API Gateway. Look!

Cool!
Notice how lambda can be triggered by the API Gateway, but it can’t interact with S3. Why not?
The reason is that we haven’t yet written a permission for the lambda role to write to S3. Let’s add that now.
Step 7: Allow the lambda role to interact with our bucket
We first have to construct the policy for interacting with S3.
// Policy for allowing Lambda to interact with S3
const lambdaS3Policy = new aws.iam.Policy(`post-to-s3-policy`, {
description: "IAM policy for Lambda to interact with S3",
path: "/",
policy: bucket.arn.apply(bucketArn => `{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:PutObject", // Very restrictive policy
"Resource": "${bucketArn}/*",
"Effect": "Allow"
}
]}`)
})Now that we created the policy, attach it to the Lambda Role.
// Attach the policies to the Lambda role
new aws.iam.RolePolicyAttachment(`post-to-s3-policy-attachment`, {
policyArn: lambdaS3Policy.arn,
role: lambdaRole.name
})Should be enough! Let’s run pulumi up and view it in the Console.

Step 8: Test it out!
Let’s POST a request in Postman and check whether it made it into S3.

Hit “Send” and visit your S3 bucket.

Step 9: Tear it down
You can just as easily destroy your stack. Simply type pulumi destroy and the entire stack will be torn down.
Next steps
You’ve now completed the entire setup — good job! There are many things you might want to consider as next steps:
- Separate your rather large file into lots of smaller components — perhaps one file for all the S3 buckets, one for the permissions, etc. — or even nested into folders for the separate functions that each cluster of resources serves
- You might want to use HTTPS instead of HTTP
- You will want to grant your lambda the permission to write logs into Cloudwatch so you can debug it more easily
- Learn how to set up different Pulumi projects and build inter-project dependencies
- Commit your project to GitHub
Here is theIndex.js file in its entirety:
"use strict";
const pulumi = require("@pulumi/pulumi");
const aws = require("@pulumi/aws");
const awsx = require("@pulumi/awsx");
const STACK = pulumi.getStack();
// Create an AWS resource (S3 Bucket)
const bucket = new aws.s3.Bucket("payloads-bucket", {
bucket: `api-meetup-bucket-${STACK}`
});
const lambdaRole = new aws.iam.Role(`role-payloads-api`, {
assumeRolePolicy: `{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow"
}
]
}
`,
})
// Policy for allowing Lambda to interact with S3
const lambdaS3Policy = new aws.iam.Policy(`post-to-s3-policy`, {
description: "IAM policy for Lambda to interact with S3",
path: "/",
policy: bucket.arn.apply(bucketArn => `{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:PutObject",
"Resource": "${bucketArn}/*",
"Effect": "Allow"
}
]}`)
})
// Attach the policies to the Lambda role
new aws.iam.RolePolicyAttachment(`post-to-s3-policy-attachment`, {
policyArn: lambdaS3Policy.arn,
role: lambdaRole.name
})
const lambdaFunction = async (event) => {
const AWS = require('aws-sdk')
const s3 = new AWS.S3()
// decode the body of the event
const payloadBuffer = new Buffer(event.body, 'base64')
const payload = payloadBuffer.toString('ascii')
const putParams = {
Bucket: process.env.S3_BUCKET, // We'll read the .env variable
Key: `${new Date().getTime()}.json`, // We'll use the timestamp
Body: payload
}
await new Promise((resolve, reject) => {
s3.putObject(putParams, function (err, data) {
if (err) {
reject(err)
} else {
resolve(data)
}
})
})
return {
statusCode: 200,
body: "Success"
}
}
const lambda = new aws.lambda.CallbackFunction(`payloads-api-meetup-lambda`, {
name: `payloads-api-meetup-lambda-${STACK}`,
runtime: "nodejs8.10",
role: lambdaRole,
callback: lambdaFunction,
environment: {
variables: {
S3_BUCKET: bucket.id
}
},
})
// create API
let apiGateway = new awsx.apigateway.API(`payloads-api-meetup-api-gateway`, {
routes: [
{
path: "/post_to_s3",
method: "POST",
eventHandler: lambda
}
]
})
// Export the name of the bucket
exports.bucketName = bucket.id
exports.apiGateway = apiGateway.urlIf you have any questions, I’d love to be helpful! Find me at adamboazbecker@gmail.com. Good luck!
