photo credit: http://www.flickr.com/photos/35068899@N03/6765038379 via http://photopin.com license

How to use Lambda for CloudFormation Custom Resources in AWS Regions that don’t support Lambda — Part 1

(Note: this article originally appeared here. I duplicated it so that it could remain published in Cloud Uprising.)

I’ll just admit my bias up front: I think that AWS CloudFormation is still one of the most useful and underrated services in the AWS repertoire, and I think that Custom Resources are one of the coolest parts of CloudFormation. With CloudFormation, infrastructure can be represented as a series of templates, which can themselves be treated for all intents and purposes as code. Much has been written on that subject so I won’t go deeper into it than that, but suffice it to say that if you’re serious about using AWS services (particularly EC2), you should be using CloudFormation.

What are Custom Resources and why should I care?

Invariably, after using CloudFormation for a while, you’ll end up needing to create something that can’t be represented as a CloudFormation template. A straightforward example might be the creation of an EMR cluster, but you may also need to put some workflow into motion or get some information that isn’t directly related to AWS.

Consider the following scenario: your organization has a pool of Elastic IP addresses (EIPs) that you can use for test workloads — these EIPs have already been specifically whitelisted by the network team (which took multiple change orders, a security review, and waiting for the firewall change window — a process you don’t wish to repeat). You have various options here; you can enter an available EIP as a parameter to the template, or you can associate the instance with an IAM role with permissions to the ec2:AssociateAddress API call, and at bootstrap have the instance run its own script to find an available EIP and associate itself with it. Both of these are fine solutions, but the first is inconvenient and the second is potentially insecure if you don’t fully trust everyone who would have access to the instances. This is just one of many potential scenarios in which a Custom Resource is the best fit.

SNS-backed Custom Resources

A Custom Resource, simply put, can take almost any input and produce almost any output, and can be implemented via either a SNS topic or a Lambda function. To set up the SNS version requires implementing the following steps:

1. Set up an SNS topic to be used by CloudFormation
2. Set up a SQS queue
3. Create a queue policy for the SQS queue to allow the SNS topic to put messages into the queue.
4. Add the SQS queue as an endpoint to the SNS topic.
5. Create an instance, preferably with an IAM profile with permissions to get and delete messages from the SQS queue. Consider setting up an Auto Scaling group for resiliency.
6. Set up a job that polls the queue, processes requests, does whatever needs to be done (for example: find an available EIP), puts a response object into S3, and deletes the message.

All of this is a bit of a pain, but of course it can be made a lot easier with a good CloudFormation template :)

Lambda-backed Custom Resources

In April 2015, AWS announced support for Lambda-backed custom resources, which is definitely a better way to do it:

• Lambda is already fully managed, so no worry about having the backend of your Custom Resource go down because you set it up incorrectly.
• For the purposes of Custom Resources, Lambda is free — you’re very unlikely to use enough resources to pass the free tier, and if you do you’re probably looking at a bill measured in pennies.

The process for setting up a Lambda-backed Custom Resource is a little different and considerably simpler:

1. Create a Lambda function that does what you want to do. I highlyrecommend “cheating” and starting with one of the functions written by AWS in one of their walkthroughs.
2. Assign a Role to the function to ensure that it has permissions required to perform the function.

That’s it! After that, you can simply use the ARN of the Lambda function as the ServiceToken in your Custom Resource.

Given the reduced cost, inherent management, and simplicity of deployment, it almost always makes sense to use Lambda for all Custom Resources rather than SNS. There’s only one problem: Lambda isn’t available in all AWS Regions — for example, if you have significant presence in Asia Pacific, you’re somewhat out of luck — Lambda is currently only available in the Tokyo region, so you’re out of luck if you wanted to implement your CloudFormation Custom Resources in Singapore, Sydney, or Beijing. CloudFormation also doesn’t currently support referencing a ServiceToken in a region other than the one in which you’re launching a CloudFormation stack (so, for example, you can’t launch a template in Singapore and reference a Lambda function in Tokyo).

Will Lambda eventually make its way to all regions? I imagine so; Lambda is an absolute game-changer in infrastructure and a critical piece of the #NoOps ideology that AWS is looking to lead, so I would imagine that it’s already slated to make its way to all regions eventually.

There is, of course, a workaround; we can create a proxy for Lambda that can be accessed via a SNS resource. In part 2, I’ll give a step-by-step on how to implement this.