How to use Lambda for CloudFormation Custom Resources in AWS Regions that don’t support Lambda — Part 2
(Note: this article originally appeared here. I duplicated it so that it could remain published in Cloud Uprising.)
(Note 2: Now that Lambda is available in all commercial regions, this article is somewhat less helpful — however, it’s still useful for a scenario in which you would need to call a Lambda function from one region while building a stack in another.)
In Part 1 of this article we covered what Custom Resources are, the difference between Custom Resources backed by SNS vs. Lambda, and why Lambda is clearly the better choice between the two. Unfortunately, at the time of this writing, Lambda is only available in a subset of AWS Regions. Furthermore, CloudFormation does not support using a Lambda function in one region as the ServiceToken for a CloudFormation template on another region (so you can’t circumvent the problem by creating your Lambda function in a supported region and referencing it in a CloudFormation template in another region). My personal guess is that AWS will eventually make Lambda available in all commercial regions, but waiting for that to happen probably isn’t a viable plan.
In the meantime, there’s an alternative; the basic idea is to create a Lambda proxy using SNS, the key component of which is an EC2 instance (or group of instances) that can poll an SQS queue for Custom Resource requests, then pass the request to a Lambda function in another region.
POC Step 1: Creating the Lambda function
As a first step in building a POC, let’s set up the Lambda function that we will be using. I’ve created a slightly modified version of AWS’s walkthrough example to find the latest Amazon Linux AMI ID and launch an instance using it, with the following modifications:
- This template doesn’t launch an EC2 instance — it’s not necessary for what we’re trying to do.
- The ARN of the Lambda function has been added to the stack outputs — we’ll use this for the proxy.
To launch the stack that creates the Lambda function, click here and accept all defaults to create the Lambda function (if you’re reading this article, I assume you’ve already got some familiarity with creating CloudFormation stacks). Note that this will default to whatever region you’re currently working in, so if you want to change it then just cancel creating the stack, change your region, and then click the link again. Obviously, whichever region you launch into must support Lambda — see here for a list of regions in which Lambda is supported
The stack should only take a couple of minutes to spin up. After it’s done, click on the “Outputs” tab and you should have two outputs. The first output (AMIID) is the AMI of the latest Amazon Linux image for your region, and the second (ServiceToken) is the ARN of the Lambda function to look up the image. Copy the ServiceToken ARN somewhere — you’re going to need it in Step 3.
POC Step 2: Creating the proxy
Now that we have a function to serve as the base, we can create the proxy. In the AWS console, navigate to a different region without Lambda support (such as Northern California), then click here to launch a proxy (note that this does launch a t2.micro instance and you may be billed accordingly). The only output from this stack is the ARN of the resulting SNS topic (ServiceToken). You should copy this for future reference as well.
While that’s spinning up, let’s talk a little bit about what this actually does. The stack goes through the process of creating a SNS-backed Custom Resource as described in part 1, but in this case the action to be taken is to just forward the request to a Lambda function (as specified by the “LambdaARN” request parameter). Fortunately, the request and response formats are identical for SNS- and Lambda-backed Custom Resources, so there’s no manipulation of the request needed — we just forward it wholesale to the Lambda function.
POC Step 3: Testing the proxy
As a means of testing the proxy, we will launch a modified version of the AWS walkthrough template that will use the SNS proxy as the basis of the Custom Resource. Ensure that you are in the same region as the SNS proxy from Step 2 and click here to launch the CloudFormation stack — use the ServiceToken values from the previous two stacks as parameters. After the stack is complete, you should have as your output the latest Amazon Linux AMI for the region in which you created the proxy.
Next steps
The above example serves as a relatively basic proof of concept of using Lambda as the backend for all Custom Resources, even for regions in which Lambda is not yet supported. A couple of obvious benefits:
- You can implement functions in Lambda and use a proxy for regions that don’t yet support Lambda — you don’t need to have a separate code base to run on EC2 Instances vs Lambda. Once Lambda gets brought to a region, you can simply implement the same function in the region and change the ServiceToken to point to the local Lambda function.
- In the near future, when Lambda can be incorporated into a user-controlled VPC, there will be lots of folks building out applications with relational databases as a backend. Custom Resources that take advantage of this type of architecture will need to live in a single region (assuming you don’t want to open your DB to the Internet). Using a proxy, you can keep the Lambda function in a single region while still using it as the backend for Custom Resources in any region.
Hopefully this has been a fun exercise and will serve as a springboard for much more complex CloudFormation templates. Please do let me know if you have any thoughts or enhancements to share.
(One final word of caution about your POC — you want to delete the last stack first; the last stack references the proxy, which in turn uses the Lambda backend. If either the proxy or the Lambda backend get torn down then the “Delete” call for the Custom Resource will fail and your stack will be stuck ,you’ll have to contact AWS support to get it removed from your CloudFormation console, which is a pain)
