Automating CI/CD with the AWS Cloud Development Kit
An introduction to automating your infrastructure as code
As businesses look to modernize their applications as part of a wider migration to Cloud, defining a consistent automated means of deploying infrastructure as code (IaC) can be a big challenge. Especially if you are trying to provide a way that’s easy for teams and squads to adopt.
The AWS Cloud Development Kit (CDK) goes a long way toward simplifying automated deployment by allowing product and development teams to create IaC in the native programming languages that they’re already familiar with — whether it’s Python, Typescript, Java, or other languages.
One of the CDK superpowers is taking the sting out of AWS CloudFormation template creation. CDK can even produce Terraform templates, if that’s your preference. In fact, Hashicorp documented a Terraform specific version of the framework.
One benefit of using CDK for creation of CloudFormation templates is the ability to distribute the templates themselves, rather than the application that produces them. There might be an external team or 3rd party vendor that have constraints deploying CDK applications. A template allows them to deploy via the CloudFormation console, Command Line Interface (CLI), or other automated manner.
Another large benefit of CloudFormation are the native AWS mechanisms to initially deploy, then maintain, infrastructure and application stacks using CloudFormation change sets.
How does CDK work?
The typical CDK lifecycle workflow is create, synthesize, test, and deploy.
Create
At its simplest level, a developer creates a CDK application in the language of their choice. My personal preference is Python. This common language is also popular among data folks who know Python well enough to maintain and expand on the codebase.
Other languages supported include:
- Javascript and Typescript, ideal for front-end developers;
- Java, suited to enterprise developers and their Java applications;
- C#; and
- Golang, which is in developer preview (at the time of this publication).
Synthesize
Once the CDK IaC application is completed, there is a CDK CLI command that allows you to synthesize the stack. That is, create the CloudFormation template from your code.
If you use Visual Studio Code, there is an excellent extension from AWS called AWS Toolkit. With this installed and enabled, every time you synthesize the stack you can view the resource stack as a tree. This allows you to inspect each resource visually, determine that the resource naming is as expected, and all resource attributes are correctly defined. It’s significantly faster than looking at a CloudFormation template.
Test
I like to “cdk synth” at regular intervals during the CDK app development so that I can spot check and correct compilation of my code. You can also create unit tests locally to ensure that given CloudFormation resources are being created as expected.
In a team environment, incorporating a test stage in the CDK CodePipeline ensures that all unit tests pass before changesets are deployed with non-compliant resources.
Standardized tests you might consider, include:
- Ensuring that all AWS resources are tagged
- Checking that security groups are not overly permissive.
The best approach to developing a comprehensive automated test framework is to start with the highest risk items — those that are very likely to occur, with a high impact if realized. Over time, work your way down to medium and lower risk items that you want to avoid via enforcement of governance.
Deploy
Once you’re satisfied with your CDK IaC application, you can use the command “cdk deploy” to deploy it into the target AWS account.
If you’re keeping things very simple, you might deploy the CDK application to a single AWS account. Each time you deploy this way, the framework produces a CloudFormation changeset and updates your target stack accordingly.
There is, however, a far more powerful and automated continuous integration, continuous delivery (CI/CD) friendly way of deploying CDK applications using CDK pipelines.
Automation with CDK pipelines
It’s always been my contention that, as a developer in a fully automated environment, all I should ever need to do is commit code to a repo and have my pull request (PR) reviewed and approved. At which point, the code should deploy when I merge it from the feature branch.
All too often this hasn’t been the case. With CDK pipelines, accomplishing this type of automation is incredibly simple and easy.
CDK pipelines supports several code repository locations, such as GitHub, BitBucket, or CodeCommit. It’s even possible to support many others by replicating commits into AWS CodeCommit and using that as the CodePipeline source.
The following code except from the CDK documentation illustrates how straight forward creating the pipeline can be.
This code illustrates the definition of a new CodePipeline using a GitHub source repository location, referencing the main branch.
The ShellStep is providing the information required to check out CDK application source code from the repository, define the CodeBuild commands needed to install the framework, then install the CDK project dependencies. The final command synthesizes the CloudFormation template from the CDK application that will be deployed into the AWS account specified by the CDK application.
Once this application is deployed into a target AWS account, the CDK based CodePipeline can then monitor the repository on the defined branch(e.g. the development branch for non-production environments, or the main branch for production environments). Whenever the pipeline detects a commit on the repo branch, it triggers a CodePipeline execution.
The CodePipeline and associated CodeBuild projects are highly configurable with multiple stages supported. Custom CodeBuild commands are also possible. In fact, you can configure all aspects of a CodeBuild project.
Adding important features (e.g. manual approvals for production stages or permission boundary changes) are very straight forward. CDK pipelines directly support these features and future articles in this series will cover this in more detail. Adding Slack channel notifications using AWS chatbot integrations is also a fairly trivial task as illustrated in the following code sample.
Because the CDK framework abstracts a lot of complexity with an extensive library of methods, features that might have previously taken weeks to develop can take less than an hour.
IaC simplified
The foremost benefit of CDK is being able to access an extensive library of methods whilst writing IaC in a language of your choice.
Whether you’re creating Identity and Access Management (IAM) users with credentials securely stored in secrets manager, defining Virtual Private Cloud (VPC) networks for Amazon EC2, or building out a fully serverless application, the extensive CDK libraries simplify these tasks compared to native CloudFormation. I’m talking about a handful of lines of code compared to a large JSON document that is difficult to manually maintain.
Perhaps even more powerful than these libraries is the ability to leverage native language features. For example, imagine you need to create three Amazon S3 buckets with similar features. In CloudFormation this would be a few dozen lines of code. In Python for CDK you could do something like:
Multiple AWS accounts
CDK supports the concept of environments, which are individual AWS accounts and regions. This makes it easy to have a common infrastructure account containing all your deployment pipelines for various stacks. You can then have each pipeline deploy the same stack to multiple AWS accounts.
This is illustrated in Fig 1.
Alternatively, a single pipeline might deploy the same stack to multiple regions in the same account as illustrated in Fig 2.
So, what are the “gotchas” of CDK?
There are always factors to be considered with any technology choice and CDK is no exception. The framework is actively developed and releases are quite frequent. This is good to know when it comes to bugs, constraints, and missing features.
Stack dependencies
One of the most common issues you’re likely to encounter are challenges with cross-stack dependencies. When you encounter them, they can be difficult to understand and often have chicken and egg conflicts between two stacks that each depend on resources from the other to be provisioned first. In short, they occur when you’re trying to reference something like an API endpoint in Stack A from a CloudFront distribution in Stack B…and Stack B has a bucket that the API in Stack A needs to reference.
By exporting the bucket output in Stack B, and importing the value into Stack A, this blocking dependency can be resolved. It can take time to understand the best approach and the way you have stacks initially designed may not be the most maintainable with the CDK method.
There are nearly always solutions to these, and similar, issues. It takes time to learn the CDK methodology and build up the knowledge and experience to either a) structure stacks to avoid the conflict or b) use export and import capabilities to mitigate the issue.
CloudFormation constraints
CDK is constrained by what’s available in CloudFormation. Much like CloudFormation, it’s possible to resort to custom resources, but this can entail a lot of time and effort depending on the complexity of the resource you’re trying to create. Your knowledge and understanding of custom resources is also a factor.
Overriding resource properties
A powerful countermeasure that CDK provides to pre-canned CloudFormation resource definitions is the ability to override any of the CloudFormation properties in the stack prior to them being created.
This is done by obtaining the underlying CloudFormation object, then overriding the property you want to modify:
While this example isn’t earth shattering, the ability to override any property with any value means that you could override EC2 instance types that aren’t available by default in the framework yet.
There are many customizations such as these that can resolve a several issues that would otherwise be serious impediments to CDK adoption. It takes time and experience to develop increasingly complex stacks in CDK and the same can be said for any IaC framework.
Broken automations
Target AWS accounts need to be bootstrapped with the CDK framework. If the bootstrap in the account gets too out of date, you need to bootstrap the account with the latest version before deployments will work again.
In automated workflows, this could cause the automation to fail until it is manually resolved.
This can be mitigated by regularly bootstrapping the account using an AWS Event Bridge rule to trigger a bootstrap event. You can even monitor the official CDK GitHub repository and update on specific release types.
Your requirements and operating model should determine whether this level of automation is worth pursuing.
Conclusion
I’ve been using CDK in production environments for the last 24 months for a variety of projects and have encountered few problems I wasn’t able to find a workaround for. Less patient developers may encounter serious impediments when resorting to unnecessary manual solutions (instead of understanding the root cause and resolving it).
The most serious issue I have encountered was unique to Version 1 of the framework. CDK Version 1 is now in a maintenance state and Version 2 didn’t have the bug. Moreover, upgrading the project to Version 2 was relatively straight forward.
If you haven’t previously considered CDK, or perhaps discounted one of the earlier versions, it’s well worth revisiting this powerful IaC framework that could reduce the barrier of entry to IaC for your application developers.
CDK lends itself to being able to rapidly develop a library of reusable, modular patterns that enforce organizational standards and best practices, like consistent resourcing tagging. Another big advantage of CDK is a completely serverless CI/CD methodology using AWS native services, such as CodePipeline.
To get started visit the AWS CDK resource pages.
Watch out for further articles in this series where we’ll dive further into some of the CDK topics introduced in this article.
Slalom is a global consulting firm that helps people and organizations dream bigger, move faster, and build better tomorrows for all. Learn more and reach out today.
