Scheduling post deployment actions as part of a CloudFormation deploy

CI Pipelines are great but CI pipelines are also vulnerable to breaking like everything else in tech.

On a recent project there was a need to deploy massively provisioned Dynamo tables for an initial “bootstrapping” of data.

This bootstrapping is done by a step function workflow which ensures the table is set to PAY_PER_REQUEST after deployment.

Trust nothing

As much as I love Step Functions I still don’t trust them 100%. I’ve used them enough to see random unforeseen errors force the entire execution to fail. Step Functions doesn’t have a “finally” block for runtime errors unfortunately.

As much as I love CI pipelines still don’t trust some random docker error not to happen while running.

And I hate to think of massively provisioned Dynamo tables just sitting there doing nothing but eating a hole in an AWS budget.

So I wanted to ensure that the CloudFormation deploy was “atomic” with a clean up task which would handle the unforeseen.

Research & Credits

It wasn’t immediately obvious to be how to do this. I thought about using the fact that “rate” cron expressions happen after X period to schedule a lambda for three hours after deploy. This would then disable it’s own trigger.

This felt ugly and the CDK code also lead to circular dependencies which tempted me to the dreaded IAM “*” escape hatch. This is when I know I’m heading down the wrong path.

So I did what any sensible dev would do. I asked Twitter 😅

Thankfully Josh Armitage and Peter Hanssens came to my help…

and a mix of step function wait state and CDK Custom Resources for AWS APIs did the trick. And it was really simple in the end….

Solution

Here‘s the solution. The deprovisioner is deployed with a 3 hour wait first and then progresses to executing the cleanup task.

About Me

An AWS Certified Solutions Architect Professional with a passion for accelerating organisations through Cloud and DevOps best practices.

If you want to work together contact me over on brianfoody.com, on LinkedIn or Twitter for a chat.