Dealing with Cross Stack Dependencies in AWS CDK
In this article I’d like to share an important insight we gained on using multiple stacks and cross stack dependencies when using AWS Cloud Development Kit (AWS CDK).
We will cover a small design flaw in our stacks having a lot of impact and causing us to suffer from a well known problem when using AWS CDK (Export StackX:***** cannot be deleted as it is in use by StackY ) way too many times.
Spoiler alert! There is no silver bullet to always avoid this problem, but by developing our stacks correctly, we were able to avoid this error for certain use cases.
I will explain this based on an example setup using an ECS Application Load Balancer and some Fargate Services being attached to it, but the general idea is applicable on different types of resources too.
Setup
Suppose we have the following two stacks, a first stack containing the load balancer:
import * as cdk from "aws-cdk-lib";
import { StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import { ApplicationListener, ApplicationLoadBalancer, ApplicationProtocol, ListenerAction } from "aws-cdk-lib/aws-elasticloadbalancingv2";
import { Vpc } from "aws-cdk-lib/aws-ec2";
interface LoadBalancerStackProps extends StackProps {
vpc: Vpc;
}
export class LoadBalancerStack extends cdk.Stack {
httpListener: ApplicationListener;
constructor(scope: Construct, id: string, props: LoadBalancerStackProps) {
super(scope, id, props);
const applicationLoadBalancer = new ApplicationLoadBalancer(this, "Alb", {
internetFacing: true,
vpc: props.vpc,
});
this.httpListener = applicationLoadBalancer.addListener("http", {
port: 80,
protocol: ApplicationProtocol.HTTP,
defaultAction: ListenerAction.fixedResponse(404, {
contentType: "application/json",
messageBody: JSON.stringify({
msg: "Default listener rule...",
}),
}),
});
}
}and a second one containing the Fargate services:
import * as cdk from "aws-cdk-lib";
import { StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import { Vpc } from "aws-cdk-lib/aws-ec2";
import { ApplicationListener, ApplicationProtocol, ListenerCondition } from "aws-cdk-lib/aws-elasticloadbalancingv2";
import { Cluster, ContainerImage, FargateService, FargateTaskDefinition } from "aws-cdk-lib/aws-ecs";
interface ServicesStackProps extends StackProps {
vpc: Vpc;
applicationListener: ApplicationListener;
services: string[];
}
export class ServicesStack extends cdk.Stack {
constructor(scope: Construct, id: string, props: ServicesStackProps) {
super(scope, id, props);
const cluster = new Cluster(this, "EcsCluster", {
vpc: props.vpc,
});
props.services.forEach((serviceName) => {
this.createService(serviceName, cluster, props);
});
}
private createService(serviceName: string, cluster: Cluster, props: ServicesStackProps) {
const taskDefinition = new FargateTaskDefinition(this, `TaskDefinition-${serviceName}`, {});
taskDefinition.addContainer(`Container-${serviceName}`, {
image: ContainerImage.fromRegistry("hashicorp/http-echo"),
portMappings: [{ containerPort: 5678 }],
environment: { ECHO_TEXT: `Hello from ${serviceName}` },
});
props.applicationListener.addTargets(`TargetGroup-${serviceName}`, {
targets: [
new FargateService(this, `FargateService-${serviceName}`, {
cluster,
taskDefinition,
}),
],
protocol: ApplicationProtocol.HTTP,
priority: 100 + props.services.indexOf(serviceName),
conditions: [ListenerCondition.pathPatterns([`/${serviceName}`])],
});
}
}Please note that these stacks have been created purely for the sake of example.
When we deploy these stacks like this:
const loadBalancerStack = new LoadBalancerStack(app, "LoadBalancerStack", {
vpc: vpcStack.vpc,
});
new ServicesStack(app, "ServicesStack", {
vpc: vpcStack.vpc,
applicationListener: loadBalancerStack.httpListener,
services: ["service-a", "service-b", "service-c"],
});we will end up with a Load Balancer having 3 services attached to it, all returning a simple hello world message.
Up to this point, everything seems fine. These 3 services are up and running and working correctly and we can make requests to all three of these services:
$ curl http://LoadBa-Alb16-XXNO7M4mHj8G-1869658469.eu-central-1.elb.amazonaws.com/service-a
Hello from service-a
$ curl http://LoadBa-Alb16-XXNO7M4mHj8G-1869658469.eu-central-1.elb.amazonaws.com/service-b
Hello from service-b
$ curl http://LoadBa-Alb16-XXNO7M4mHj8G-1869658469.eu-central-1.elb.amazonaws.com/service-c
Hello from service-cSo far, so good.
Problem
Now, although we have a setup in which we can easily add and update services, let’s say we want to delete one of them. Since we pass in our services dynamically, that shouldn’t be an issue. Suppose we want to remove service-c , we could just remove it from the list and redeploy our changes and be done with it.
const loadBalancerStack = new LoadBalancerStack(app, "LoadBalancerStack", {
vpc: vpcStack.vpc,
});
new ServicesStack(app, "ServicesStack", {
vpc: vpcStack.vpc,
applicationListener: loadBalancerStack.httpListener,
services: ["service-a", "service-b"], // <== "service-c" removed here
});Although this is not an issue code wise since we just remove one element, we will run into issues in CloudFormation:
If you worked with AWS CDK and multiple stacks before, you probably ran into this issue already at some point. The problem here is that when a stack has a dependency on another stack, it will result in two CloudFormation stacks using CloudFormation exports & imports to pass values from one stack to the other.
In this specific case, because we add a target group to our Load Balancer in the LoadBalancerStack , which is then being used in the ServicesStack , is causing this target group ARNs to be exported. The fact that Target Group ARNs need to be exported here are part of the underlying problem, but we’ll get to that later.
Although we already know this import in our downstream stack will no longer be used, there is no way to tell CloudFormation to deal with this since stacks will always be updated individually.
Even if we would try to update our services stack first, it would still result in the load balancer stack being included in the update since it has a dependency on it. So at this point we are stuck.
Solution #1
A first solution we came up with, which can also be found all over the internet, and is also explained in the Official Documentation from AWS, is to delete the resources but temporarily keep this export instead. Since we do know it won’t be used anymore, no harm can be done keeping a non existing ARN in place as an export.
To do so, we could put the CloudFormation output explicitly in our CDK stack by taking the exact key, value and export name from CloudFormation. Note that these values have to be exactly the same, otherwise it would result in an update suffering the same issue.
new cdk.CfnOutput(loadBalancerStack, "ExportsOutputRefAlbhttpTargetGroupservicebGroupA87A6DF0022FEF46", {
value: "arn:aws:elasticloadbalancing:eu-central-1:723071428250:targetgroup/LoadBa-Albht-JLLUIYZXFEF3/0499a28d95593ebf",
exportName: "LoadBalancerStack:ExportsOutputRefAlbhttpTargetGroupservicebGroupA87A6DF0022FEF46",
});If we deploy the changes with this output in place, deployment will succeed and the service will be removed.
This solution is obviously not ideal for our use case, and also has an additional downside that we would need a second deployment to remove the no longer used export.
It would of course also be possible to remove the services stack completely first, but in the real world you probably don’t want to do that.
Important: Please note that when you have a use case in which you have a resource in an upstream stack being referenced in a downstream stack that needs to be deleted, this is probably the only possible solution. However, for this specific use case here, a better approach for deploying these resources exist!
Problem: Deep dive
Let’s take a closer look at the actual problem. Our LoadBalancerStack only contains our actual Load Balancer and one listener being added to it:
import * as cdk from "aws-cdk-lib";
import { StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import { ApplicationListener, ApplicationLoadBalancer, ApplicationProtocol, ListenerAction } from "aws-cdk-lib/aws-elasticloadbalancingv2";
import { Vpc } from "aws-cdk-lib/aws-ec2";
interface LoadBalancerStackProps extends StackProps {
vpc: Vpc;
}
export class LoadBalancerStack extends cdk.Stack {
httpListener: ApplicationListener;
constructor(scope: Construct, id: string, props: LoadBalancerStackProps) {
super(scope, id, props);
const applicationLoadBalancer = new ApplicationLoadBalancer(this, "Alb", {
internetFacing: true,
vpc: props.vpc,
});
this.httpListener = applicationLoadBalancer.addListener("http", {
port: 80,
protocol: ApplicationProtocol.HTTP,
defaultAction: ListenerAction.fixedResponse(404, {
contentType: "application/json",
messageBody: JSON.stringify({
msg: "Default listener rule...",
}),
}),
});
}
}Why are we running into issues with this stack when we are removing a service from our services stack?
The actual problem lies in the fact that we are passing our httpListener: ApplicationListener from this stack to our services stack and use it there to add target groups for our services:
props.applicationListener.addTargets(`TargetGroup-${serviceName}`, {
targets: [
new FargateService(this, `FargateService-${serviceName}`, {
cluster,
taskDefinition,
}),
],
protocol: ApplicationProtocol.HTTP,
priority: 100 + props.services.indexOf(serviceName),
conditions: [ListenerCondition.pathPatterns([`/${serviceName}`])],
});Although this code resides in our ServicesStack it is effectively modifying resources from our LoadBalancerStack and that is the root cause of our issue. This becomes very clear when performing a diff after removing a service.
As we can see here, changes will be made to both stacks, the AWS::ElasticLoadBalancingV2::TargetGroup and AWS::ElasticLoadBalancingV2::ListenerRule belong the the load balancer stack, while the other resources belong to the services stack. The fact that these changes spread across multiple stacks, is because we modify resources from another stack, the applicationListener in this case.
It’s not really described as a strict rule in CDK best practices, but I believe that modifying resources in one stack’s CDK code, should never modify resources belonging to other stacks.
Solution #2
Now that we understand the problem, it’s time to move on to a proper fix for this issue. We would like the target groups and listener rules to be scoped properly. Each construct when being created has it’s scope:
constructor(scope: Construct, id: string);and it is key for this scope to be correct.
If we take a look at our current setup, we can see that the target group and listener rule resources belong to the load balancer stack’s scope, and that is not what we want. We didn’t even realise this at first.
The reason for this is that the target group gets added behind the scenes when calling ApplicationLoadBalancer.addListener and will live in the load balancer’s scope.
For getting this fixed, we should not modify resources from the load balancer stack directly, but instead, we should import the listener into our services stack. To do so, we can make use of ApplicationListener.fromApplicationListenerAttributes which will create a construct in the scope of our current stack. If we then add the target group to this imported listener, it will also be scoped to our service stack instead. And that is exactly what we want to achieve!
ApplicationListener.fromLookupexists too, but this will perform a lookup at synthesis time and therefore can only deal with static values. This is not suitable for our use case since our load balancer reference will be a token.
We can use this construct by specifying the listener’s ARN and a reference to the Security Group. The listener is referenced via an ARN anyway, for the security group we can use a similar mechanism to import that too. Otherwise, we might end up with cyclic references on the security group.
const applicationListener = ApplicationListener.fromApplicationListenerAttributes(this, "ApplicationListenerImport", {
listenerArn: props.applicationListener.listenerArn,
securityGroup: SecurityGroup.fromSecurityGroupId(this, "SecurityGroupImport", props.albSecurityGroup.securityGroupId),
});Note that the securityGroup is a required property for the import, so we had to make a small change to our load balancer stack to create the security group ourselves.
After these changes our two stacks look like this:
import * as cdk from "aws-cdk-lib";
import { StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import { ApplicationListener, ApplicationLoadBalancer, ApplicationProtocol, ListenerAction } from "aws-cdk-lib/aws-elasticloadbalancingv2";
import { Peer, Port, SecurityGroup, Vpc } from "aws-cdk-lib/aws-ec2";
interface LoadBalancerStackProps extends StackProps {
vpc: Vpc;
}
export class LoadBalancerStack extends cdk.Stack {
httpListener: ApplicationListener;
securityGroup: SecurityGroup;
constructor(scope: Construct, id: string, props: LoadBalancerStackProps) {
super(scope, id, props);
this.securityGroup = new SecurityGroup(this, "SecurityGroup", {
vpc: props.vpc,
});
this.securityGroup.addIngressRule(Peer.anyIpv4(), Port.tcp(80));
this.securityGroup.addIngressRule(Peer.anyIpv6(), Port.tcp(80));
const applicationLoadBalancer = new ApplicationLoadBalancer(this, "Alb", {
internetFacing: true,
vpc: props.vpc,
securityGroup: this.securityGroup,
});
this.httpListener = applicationLoadBalancer.addListener("http", {
port: 80,
protocol: ApplicationProtocol.HTTP,
defaultAction: ListenerAction.fixedResponse(404, {
contentType: "application/json",
messageBody: JSON.stringify({
msg: "Default listener rule...",
}),
}),
});
}
}import * as cdk from "aws-cdk-lib";
import { StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import { SecurityGroup, Vpc } from "aws-cdk-lib/aws-ec2";
import {
ApplicationListener,
ApplicationProtocol,
ApplicationTargetGroup,
IApplicationListener,
ListenerCondition,
} from "aws-cdk-lib/aws-elasticloadbalancingv2";
import { Cluster, ContainerImage, FargateService, FargateTaskDefinition } from "aws-cdk-lib/aws-ecs";
interface ServicesStackProps extends StackProps {
vpc: Vpc;
applicationListener: ApplicationListener;
albSecurityGroup: SecurityGroup;
services: string[];
}
export class ServicesStack extends cdk.Stack {
constructor(scope: Construct, id: string, props: ServicesStackProps) {
super(scope, id, props);
const cluster = new Cluster(this, "EcsCluster", {
vpc: props.vpc,
});
const applicationListener = ApplicationListener.fromApplicationListenerAttributes(this, "ApplicationListenerImport", {
listenerArn: props.applicationListener.listenerArn,
securityGroup: SecurityGroup.fromSecurityGroupId(this, "SecurityGroupImport", props.albSecurityGroup.securityGroupId),
});
props.services.forEach((serviceName) => {
this.createService(serviceName, cluster, applicationListener, props);
});
}
private createService(serviceName: string, cluster: Cluster, applicationListener: IApplicationListener, props: ServicesStackProps) {
const taskDefinition = new FargateTaskDefinition(this, `TaskDefinition-${serviceName}`, {});
taskDefinition.addContainer(`Container-${serviceName}`, {
image: ContainerImage.fromRegistry("hashicorp/http-echo"),
portMappings: [{ containerPort: 5678 }],
environment: { ECHO_TEXT: `Hello from ${serviceName}` },
});
applicationListener.addTargetGroups(`TargetGroups-${serviceName}`, {
targetGroups: [
new ApplicationTargetGroup(this, `TargetGroup-${serviceName}`, {
vpc: props.vpc,
targets: [
new FargateService(this, `FargateService-${serviceName}`, {
cluster,
taskDefinition,
}),
],
protocol: ApplicationProtocol.HTTP,
}),
],
priority: 100 + props.services.indexOf(serviceName),
conditions: [ListenerCondition.pathPatterns([`/${serviceName}`])],
});
}
}If we deploy the stacks like this, we will end up with target groups and listeners rules being correctly created in the scope of the services stack instead.
If we take a look at the deployed CloudFormation stack, we see that the listener rules are now present as a resource for the imported application listener, and the target groups are created within the services stack.
If we look at the load balancer stack now, we see only the Load Balancer and the http listener are present there.
And only the Load Balancer’s ARN and the Security Group’s id are being exported.
We now have a proper segregation between our load balancer resources and the resources being service specific. This way we can easily create, update and delete services without modifying a single resource from a different stack.
Conclusion
When working with multiple stacks in AWS CDK, pay attention to:
- Not modify resources from stack X from within stack Y
- Correctly scope your constructs
- Use imports or lookups if you want to make use of resources from other stacks
If you really need to remove resources being actually exported and imported between different stacks, you will have to follow the two step removal process.
