Short feedback cycles on AWS Lambda

A Makefile that enables to iterate quickly

What is AWS Lambda?

AWS Lambda is a compute service that runs your code in response to events […]

You configure the events that trigger the execution of your code. This can be an HTTP request on an API gateway, a message on an SNS topic, a bucket event from S3, etc.

You ship your code in one of three forms: 1) literally, the source code 2) a container image or 3) an executable. You configure the entrypoint that accepts the event data, and from there on, your code can do its thing.

Simple Lambda triggered by S3 “object create” events

How to develop for AWS Lambda — the easy way

The most straightforward way to develop AWS Lambda functions is directly in the AWS console.

Writing code directly in the AWS Console

You write code in an in-browser editor, define a test payload, deploy and test by clicking two buttons. Your feedback cycle is about 5 seconds.

While this is fast and good enough for a lot of use cases, there is a couple of drawbacks with this approach:

• You can only do this for the supported languages, at this moment “the console code editor supports only Node.js, Python, and Ruby”.
• Your code is not checked in to version control.
• You are not coding in the full-fledged IDE that you’re used to, with your personal extensions, linters, formatters, etc.
• The only testing you do is using the test payloads.

How to develop for AWS Lambda — the Makefile way

If you don’t use the in-browser code editor, you have to package and upload your code. We’ll do it as frictionless as possible using a Makefile. In this example, we write Go code locally, build a Go binary, zip it, upload it and deploy it on AWS Lambda, but the same approach works for other packaging methods too. For source code, you’ll also need to zip your libraries, and for containers you’ll have to build and push the container.

Of course, having code locally opens up a whole world of testing locally, without even deploying to AWS lambda. This is even faster, but at some point you’ll want to try out your deployed lambda e.g. to verify that its role has all the right permissions.

As soon as you want to do the “build → deploy → test → logs” loop as fast as possible, this Makefile approach can help.

The first step is to find test payloads that you know your lambda should be able to handle. Save them as .json files and put them in a folder next to your lambda code. We’ll use those to trigger the freshly deployed lambda.

For example, if your lambda consumes HTTP GET requests, you would save something like the following as lambda-test-payloads/type1.json :

{
  "version": "2.0",
  "routeKey": "$default",
  "rawPath": "/documents",
  "rawQueryString": "user_id=4&query=dataminded",
  "headers": {
    "x-forwarded-port": "443",
    "x-forwarded-for": "xx.xx.xx.xx",
  },
  "queryStringParameters": {
    "user_id": "4",
    "query": "dataminded"
  },
  "isBase64Encoded": false
}

I usually obtain those by having the lambda print out its incoming request, and then copy pasting from Cloudwatch logs, plus a bit of swapping single quotes to double quotes and uppercase False to lowercase false to make it valid json. If cat type1.json | jq doesn’t complain, you’re probably good. You can also strip out the parts of the json that you’re sure your lambda won’t need, in my case the whole requestContext.

The feedback you want is whether your lambda succeeded, and the execution logs. For the latter, there are two options: either 1) get the latest 4KB of logs from the response of the trigger or 2) stream the tail of the logs in a separate terminal.

So I spent some time figuring out how to do that while making sure it’s really the latest code running (that was the hard part), and this is the Makefile I ended up with. My feedback loop takes ~15 seconds:

LAMBDA_FUNCTION_NAME = "my-function"
LAMBDA_TEST_PAYLOAD_FILE = "lambda-test-payloads/type1.json"
#LAMBDA_TEST_PAYLOAD_FILE = "lambda-test-payloads/type2.json"

# build the go binary
build:
 @echo "Building..."
 @# executable needs to be named "bootstrap" for AWS Lambda al2023
 @# we build for arm64 to match the AWS Lambda environment
 GOOS=linux GOARCH=arm64 go build -o bin/bootstrap .

# upload to aws
deploy-lambda:
 @echo "Deploying to AWS Lambda..."
 @zip -j bin/bootstrap.zip bin/bootstrap
 aws lambda update-function-code \
  --function-name $(LAMBDA_FUNCTION_NAME) \
  --zip-file "fileb://bin/bootstrap.zip" \
 | cat
 aws lambda wait function-updated-v2 --function-name $(LAMBDA_FUNCTION_NAME)
 @echo "New CodeSha256"
 @aws lambda get-function --function-name $(LAMBDA_FUNCTION_NAME) | jq '.Configuration.CodeSha256'

# attach a version to the lambda based on the commit hash
publish-lambda-version:
 @aws lambda publish-version --function-name $(LAMBDA_FUNCTION_NAME)

describe-lambda:
 @aws lambda get-function --function-name $(LAMBDA_FUNCTION_NAME)

# invoke the lambda
trigger-lambda:
 @echo "Invoking the lambda..."
 aws lambda invoke \
  --no-cli-pager \
  --payload fileb://$(LAMBDA_TEST_PAYLOAD_FILE) \
  --function-name $(LAMBDA_FUNCTION_NAME) \
  --log-type Tail \
  /dev/null \
  | jq -r '.LogResult' \
  | base64 --decode
 # Note: only last 4KB of logs are returned by the API

# tail of all log events (run in a separate terminal)
stream-lambda-logs:
 @aws logs tail /aws/lambda/$(LAMBDA_FUNCTION_NAME) --follow

# run the whole dev cycle
dev-cycle: build deploy-lambda trigger-lambda

Then write your code and run make dev-cycle, which builds the binary, uploads it to aws lambda, triggers it and prints the result back to the terminal.

Have fun writing lambdas!

Thanks to Jos Teunissen & Jonathan Merlevede for their proofreading and constructive feedback.