How we almost lost our cloud databases

We have lots of projects using Amazon Cloud Development Kit to manage our infrastructure.

In order to support all of these projects we have developed our own corporate CDK construct library that is used by all of our projects.

We try to keep all of our projects up to date and have them use the latest corporate CDK construct library version, but as you can imagine, not all projects are always updated at the same time. Especially the more dormant projects are often put on the back burner and aren’t updated as often as they should.

Now while working on our construct library to support a new feature we noticed some deprecated code in the AWS CDK library.

The instanceProps had become deprecated and being the good forward-looking developers that we are we decided to remove the deprecated code

Previously, the code to roll out an Aurora instance looked like this

declare const vpc: ec2.Vpc;
const cluster = new rds.DatabaseCluster(this, 'Database', {
  engine: rds.DatabaseClusterEngine.auroraMysql({
    version: rds.AuroraMysqlEngineVersion.VER_3_03_0,
  }),
  instances: 2,
  instanceProps: {
    instanceType: ec2.InstanceType.of(ec2.InstanceClass.BURSTABLE3, ec2.InstanceSize.SMALL),
    vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
    vpc,
  },
});

The new of doing it involves using the writer and readers properties

const cluster = new rds.DatabaseCluster(this, 'Database', {
  engine: rds.DatabaseClusterEngine.auroraMysql({
    version: rds.AuroraMysqlEngineVersion.VER_3_03_0,
  }),
  vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
  vpc,
  writer: rds.ClusterInstance.provisioned('Instance1', {
    instanceType: instanceProps.instanceType,
    isFromLegacyInstanceProps: instanceProps.isFromLegacyInstanceProps,
  }),
  readers: [
    rds.ClusterInstance.provisioned('Instance2', {
      instanceType: instanceProps.instanceType,
      isFromLegacyInstanceProps: instanceProps.isFromLegacyInstanceProps,
    }),
  ],
});

Easy enough right ?

Well not quite…. while doing a cdk diff to see if our new construct library was still compatible with existing projects we noticed the following :

❯ npm run cdk diff

> cdk-simple-aurora@0.1.0 cdk
> cdk diff
Stack CdkSimpleAuroraStack
Resources
[-] AWS::RDS::DBInstance myproject/Instance1 myprojectInstance1F6D495B2 destroy
[+] AWS::RDS::DBInstance myproject/Instance1 myprojectInstance1ED45AF0B
[~] AWS::Lambda::Function Custom::VpcRestrictDefaultSGCustomResourceProvider/Handler CustomVpcRestrictDefaultSGCustomResourceProviderHandlerDC833E5E
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 4b996a3e5a083d5c78c6f30a8571a94fb7ec557eecbe54dbc065faba0d9076e6.zip
 │       └─ [+] 7f18a11296f35510ee16538afec983ed6312e12afbf81b777089a9f8e34e2474.zip
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.4b996a3e5a083d5c78c6f30a8571a94fb7ec557eecbe54dbc065faba0d9076e6
         └─ [+] asset.7f18a11296f35510ee16538afec983ed6312e12afbf81b777089a9f8e34e2474
[~] AWS::RDS::DBCluster myproject myproject02321A7C
 └─ [~] DeletionProtection

✨  Number of stacks with differences: 1

Notice how this diff showed that if we were to deploy our project with this new construct library, our RDS database instance would be destroyed and re-created.

a database system burning down to the ground (Stable Diffusion XL 1.0)

By introducing the isFromLegacyInstanceProps:true property in our CDK project we can ensure that our RDS database will no longer get removed and re-created

❯ npm run cdk diff

> cdk-simple-aurora@0.1.0 cdk
> cdk diff
Stack CdkSimpleAuroraStack
Resources
[~] AWS::Lambda::Function Custom::VpcRestrictDefaultSGCustomResourceProvider/Handler CustomVpcRestrictDefaultSGCustomResourceProviderHandlerDC833E5E
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 4b996a3e5a083d5c78c6f30a8571a94fb7ec557eecbe54dbc065faba0d9076e6.zip
 │       └─ [+] 7f18a11296f35510ee16538afec983ed6312e12afbf81b777089a9f8e34e2474.zip
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.4b996a3e5a083d5c78c6f30a8571a94fb7ec557eecbe54dbc065faba0d9076e6
         └─ [+] asset.7f18a11296f35510ee16538afec983ed6312e12afbf81b777089a9f8e34e2474
[~] AWS::RDS::DBCluster myproject myproject02321A7C
 └─ [~] DeletionProtection

✨  Number of stacks with differences: 1

Now off course it is possible (and recommended) to enable deletion projection on your RDS databases, and it is something that we enabled by default on all of our databases, both for non-production and production, so the impact here would be limited to having to deal with failing CDK deployment pipelines.

a modern photorealistic representation of a database with a big chain and lock to protect it (Stable Diffusion XL 1.0)

But it just goes to show how important it is to have this deletion projection enabled AND also verify your CDK changes before rolling them out.

Conclusion

It’s obviously a good idea to try and stay ahead of the curve and use the latest and greatest libraries at your disposal, but do take into account that more often than not a lot more effort is needed then simply bumping a version.

Imaging if you don’t have delete protection enabled. This seemingly harmless change in your CDK code could all of a sudden result in you loosing your entire database.

So the key take-aways here are

• Always verify your CDK upgrades / changes in a non production environment by performing a cdk diff before your start rolling out your changes with cdk deploy. This will compare the CloudFormation templates generated by your CDK project with the resources that are currently deployed on your actual AWS account.
• Enable deletion protection on your databases. We rarely remove and re-create RDS instances simply because it takes a long time to do it, and you lose all of your data. So that protection would have resulted in the cdk deploy to fail