Part 2: Advanced Custom Resources with AWS CDK
Custom Resources with AWS CDK
Using AWS CloudFormation or CDK, you might find yourself needing to create a resource that is not available under CloudFormation resource types.
In this post, you will learn how to use CDK to create an S3 object with AWSCustomResource and walk you through the life-cycle of your resource. We will also take a look at some pitfalls and gotchas that I encountered working with CDK AWSCustomResource.
I assume that you are somewhat familiar with CDK, but if not, I highly recommend giving it a try. You can go through the CDK Getting started guide from AWS to get going.
If you just want to see how it works, head to the working example in GitHub and simply follow the instructions in README.MD.
What are AWS Custom Resources?
Custom Resources allow you to write custom logic in your CloudFormation deployment. You implement the creation, update, and deletion logic to define the custom resource deployment.
CDK Implements the AWSCustomResource — a lambda-backed custom resource that uses the AWS SDK to provision your resources. This means that the CDK stack deploys a “provisioning lambda” which, upon deployment, calls the AWS SDK APIs that you defined for the resource lifecycle (create, update and delete).
The Problem
Assume you are deploying your cloud stack which includes an S3 bucket. Being a CloudFormation resource type, the bucket is deployed, updated, and destroyed naturally. Now, for our example, assume that you also want to create some objects in your S3 bucket for a system configuration. How will you do that? You could naively create them using boto3, but this means that they will not be destroyed with your stack. Furthermore, they could prevent your stack from being destroyed when needed.
Custom Resources to the rescue!
Prerequisites
- Access to an AWS account in which you can deploy your custom resources. Don’t forget to destroy your resources when you are done!
- A clean CDK project (you can do this) — I will use Python in this example.
- Ability to work with a CDK Python project in a virtual environment.
Let’s get to work
We will start by creating a nice construct to host all this custom goodness, some of the imports will be used later.
We created our own construct to wrap the creation of the custom object and make it friendlier. I exposed the log_retention of the custom resource, but it is totally optional and you can decide what you want to expose in AWSCustomResource.
We also referenced the bucket from its bucket name so we can get its arn. You can of course simply pass the arn, but this way you can reference external buckets by name.
Moving on, we need to define the creation, update, and deletion of our resources. In the case of the S3 object, “update” is merely uploading a new object, so create and update are the same.
This code goes in the construct init function:
And the implementation is in the following member functions:
To understand how to implement the lifecycle events above, you need to turn to the AWS SDK API Reference. Since we want to create, update, and delete an S3 object, we need to look at the docs for S3.putObject and S3.deleteObject, where you can find the description of the parameters that you need to pass to it along with the API name (putObject/deleteObject) and the service name (S3).
IMPORTANT: For the physical resource id, you need to use something that will not change over deployments. I simply used the object full name which is unique and unchanging. If your physical id changes your resource will be deleted on an update, since CloudFormation considers these different resources.
Next, we will create the policy, role, and the actual custom resource:
Let’s break this down:
First, we define the policy that will be added to the provisioning lambda role. CDK docs are a little unclear about this saying:
The policy to apply to the resource.
This policy will actually be attached to the provisioning lambda role. A documentation issue was created and I expect the docs to be updated soon.
We let CDK decide what permissions are needed for the SDK API, by calling the from_sdk_calls, and set the resource to the exact object name which is just what we need.
IMPORTANT UPDATE: Note that specifying the exact arn for from_sdk_calls means that if you try to update your resource name, update will fail due to missing permissions. The only way I managed to overcome this is to provide lambda permissions so that it will be able to create and delete the new and old resources. See this CDK issue for details.
Next, we create the role of the provisioning lambda, specifying lambda as the service principal, and attaching the managed policy for basic lambda execution. This will allow this lambda to write logs that can help us troubleshoot deployment issues. You can pass None as the role, but then you get no logs.
It is important to keep in mind, that the lambda role is one for all custom resources that you create in the stack (see the docs).
In our example, it doesn’t matter, but if your custom resources require special policies, then you will overwrite the previous ones if you create different roles.
And finally, we create our custom resource:
I passed the default timeout just to stress that it’s there, if your resource requires, you can pass something else.
IMPORTANT UPDATE: I recently experienced a condition, where creating your own role can cause issues. For example, when you’re using a CDK construct that uses AWSCustomResource itself, like in this issue. You shouldn’t worry too much if you’re just giving your Lambda permissions to log, but sometimes you need to give it special permissions in which case it can cause your permissions to go missing. You can instead avoid passing a role to the resource and instead call grant_principal.add_to_policy. I used the following code to add iam:PassRole when creating an IoT Role Alias as a custom resource:
Putting it to the Test
You are now ready to deploy, update, and destroy your stack. You can follow the instructions here, and make sure to check out your S3 object, as it is created, updated, and eventually deleted.
Failing Stacks
If for some reason, your deployment is failing, you might experience a hanging stack deployment. This can occur if you have a problem with your lifecycle events like using the wrong API (create) for the on_delete event. this means that CloudFormation cannot remove this resource and will hang there for a long time.
You can get rid of it manually by following this AWS guide, but I was also able to delete the stack by clicking the delete button, waiting for the stack to fail, and then delete again, checking the “Retain resource” which will allow the stack to be deleted. You can later manually delete the remaining resources.
Wrapping Up
In a world of automation and infrastructure-as-code, we must make sure that the resources we deploy have a well-defined life-cycle.
AWS Custom Resources can be used to deploy resources with CloudFormation that are not supported otherwise. This will keep your deployment consistent and without “rogue” resources, which management can later cost you a lot of time.
With CDK, you can use the example above as the basis for creating your own custom resources as part of your CloudFormation deployment.
Updates
- It seems that there’s an issue where using ElasticSearch Domain logging with log groups may cause the singleton role to be overwritten.
