The Tale of AWS CDK Refactoring, Logical IDs, and Lost Resources

Photo by George Pastushok on Unsplash

Hello fellow CDK developers!

I love working with AWS CDK, mainly because it is code, which has many benefits.

As engineers, we take our code seriously. We nurture it and make sure it stays clean. A big part of this is fearlessly refactoring (testing got your back, right?).

I would like to share with you how important it is to be aware of logical IDs, what happens when they change and how it relates to your code structure (Spoiler: Your resources are deleted and then re-created).

This blog assumes knowledge of the AWS CDK framework and CloudFormation.

AWS CDK Construct IDs and CloudFormation Logical IDs

AWS CDK constructs have an ID that must be unique in its scope.

For example:

• This will cause a synth time error due to ID conflict:

bucket1 = Bucket(scope=self, id=’Bucket’, …)
bucket2 = Bucket(scope=self, id=’Bucket’, …)

• This is fine since they are under a different parent construct and have a unique path:

class Parent1(Construct):
   ...
   bucket = Bucket(scope=self, id=’Bucket’, …)
class Parent2(Construct):
   ...
   bucket = Bucket(scope=self, id=’Bucket’, …)
parent1 = Parent1(scope=self, id='Parent1', ...)
parent2 = Parent2(scope=self, id='Parent2', ...)

The underlying CloudFormation resources are assigned logical IDs that must be unique on the stack level. AWS CDK generates the ID based on the construct ID and hierarchy as you can see in the CloudFormation console:

Logical IDs in CloudFormation

Note how some resources are set to be deleted and others to be retained. In production, you should have all of your stateful resources retained and backed up.

Initial CDK stack

And here’s how it looks in the CloudFormation console with the new tree view that shows the AWS CDK stack hierarchy:

CloudFormation tree view of the Data Platform stack

See AWS CDK’s docs to learn more about AWS CDK IDs.

When Does a Logical ID Change?

When working with AWS CDK, there are two cases in which a resource logical ID would change:

1. You change its construct ID — this can easily be avoided with little awareness.
2. You change the construct hierarchy and thus change its CDK path and CloudFormation logical ID.

What Happens When a Logical ID Changes?

This is the important part, so listen carefully:
When a logical ID changes for a construct, then this construct is deleted and re-created as a new resource.

This can be your precious DB table, which can seriously put a damper on your weekend. While your database must have a retain deletion policy in production plus backups, I still wouldn’t like to get a surprise “Migrate your users’ data to another table ASAP!”

For example, your retained DynamoDB table will become an “orphan” when its logical ID changes after refactoring this code - a new table will be created and the old one will no longer belong to the stack.

Now, let’s do some refactoring:

In UserManagement:

• Create another table for user permissions.
• Move all tables under a new “UsersData” construct.

In DataLake:

• Move all buckets under a LakeBuckets parent construct.

Data Platform refactored using constructs

You can run cdk diffto see how your weekend is going to be ruined, notice how your old buckets are deleted, how the table is now an orphan and new resources are created instead:

Stack resources being replaced

Let’s deploy our updated code and see what happens in CloudFormation. We get a nice hierarchy of new resources.

Data Platform refactored using constructs

Here are the relevant stack events which show that new resources were created (all with different logical IDs):

Stack events after construct refactoring

Photo by GeoJango Maps on Unsplash

Pinning Logical IDs

One option to avoid such a thing is to pin the logical ID of your stateful constructs by calling the overrideLogicalId function available for any AWS CDK construct.

Simply grab the current logical ID of your construct and call:

bucket = Bucket(scope=self, id=’Bucket’, …)
bucket.node.default_child.override_logical_id('CURRENT_LOGICAL_ID')

In the code below, I used the existing logical ID so that resources are not recreated (hence the random ID suffix):

CDK Stack refactoring — overriding logical IDs

Let’s run cdk diff again and see that only metadata is changed this time:

CDK diff when overriding logical IDs

And here’s how it looks in CloudFormation, see that the correct hierarchy is shown thanks to the CDK metadata changes, but with our original resources:

CloudFormation tree view when overriding logical IDs

AWS CDK Refactoring

As a general rule of thumb, you should never update your stateful resources’ logical ID. This is easier said than done - how can we do this and still feel free to refactor our code freely?

This AWS CDK-RFC page talks exactly about this and I will update on any interesting developments. But for the moment, let’s see how we can approach refactoring CDK code.

Construct Based Structure

Usually, when creating a CDK app, you encapsulate related resources within parent constructs. For example, in the above example, where I created a UserManagement(Construct) and kept all of the user management resources.

This results in the following:

• Any construct’s path and logical ID will include its parents’ path.
• This will also help you visualize in CloudFormation tree view or other visualization tools.
• But also: Since the hierarchy is included in the logical ID, moving resources around will result in them being deleted and re-created (new constructs are first created and then the old ones are deleted).

Non-Construct Based Structure

You can also take a different approach: Using simple classes that do not inherit Construct and thus do not participate in this game of logical IDs.

I have made the following code changes:

• UsersData and DataBuckets do not inherit Construct anymore.
• I still pass the parent scope so that I can pass it to the child resources instead of “self”.

CDK refactoring with simple classes

running cdk diff , we see that nothing was recreated, but also that we lost the hierarchy benefits:

CDK diff under non-construct refactoring

Pros:

• You get the benefit of clean code and you can structure your application the way you want.
• You have zero logical ID changes (unless you change a construct ID yourself).

Cons:

• You lose the CDK hierarchy and can no longer visualize your stack in tree view or other tools — your CloudFormation stack looks just like before the refactoring.
• The normal, non-construct classes “contaminate” the IDs namespace of the construct they live in: since your constructs are flat, you cannot have more than one ID with the same value in two classes under the same construct.

Which Option is Right for Me?

Given the above options, you can weigh the pros and cons and decide for yourself, but let me share my perspective.

I believe in something I’d like to call “Top Level Domain Structuring”. This means that you only create constructs based on your top-level domains. This makes it is highly unlikely that you would move a database for example between each other. And in each domain, use simple classes for code structuring and encapsulation.

A library, for example, needs to be one domain - it has to be a construct to avoid contaminating the user’s ID space, but internally should include simple classes. This will also prevent breaking your users’ resource IDs if you move things around internally.

This will let you enjoy both worlds - have the CloudFormation structure and hierarchy, but also decide how deep you want it to go. My rule of thumb is to keep it on a “Top Level Domain” level.

This option is not a silver bullet because you might still need to move things around between constructs, or add parent constructs but in this case, you can override logical IDs as a backup plan.

AWS CDK Testing

AWS CDK 2 added the ability to easily test your stack, and I highly recommend giving it a try. This can help you verify that nothing breaks for your critical resources. For example, I decided to verify that the logical ID for my DynamoDB table does not change or that it is under a specific, known hierarchy.

To learn more see “Testing Constructs”.

AWS CDK Refactoring and Logical IDs: A Wrap-Up

When working with AWS CDK code, it is important to know what logical IDs are, what can make them change, and what happens when they do change (your resource will be recreated).

Armed with this knowledge, you should have a strategy for making code changes and know how to avoid such resource recreation.

You can also use CDK testing and CDK diff to help you detect such unexpected changes.

Thanks to Amir Zahavi for getting me to think of non-construct classes.

Thanks to Alex Pulver for the long talks and ever-helpful thoughts and insights.