IaC — Automate your infrastructure with AWS and CDK
Many teams struggle to deploy solutions that can be reproduced in multiple environments, due to the complexity of creating their infrastructure in the cloud. In the past, creating the infrastructure needed for your service could have been very complicated, but with the introduction of IaC, the ability to automate and replicate your infrastructure became possible. In this article, we will discuss how we at Zigbang use IaC, together with AWS and CDK, to deploy our services inside our microservice architecture.
But after all, what is IaC?
IaC stands for Infrastructure as Code, and as the name says, it represents the process of defining the infrastructure of a service by the instruction contained in an executable file. By definition, it allows us to create the infrastructure needed that can not only be programmable but most importantly, that can also be versioned and managed by its developers. There is a variety of proprietary solutions available in the market for platforms such as Azure, Google Cloud, Amazon AWS, and others.
Most of the available solutions for IaC are based on structured objects that can be translated and interpreted by your cloud platform, for example, using JSON, YAML, and other formats. More recently though, we started seeing the movement to create tools for developers and dev-ops engineers to develop their infrastructure code in their chosen programming language. This idea is very powerful because it truly enables programmability and reusability inside of your infrastructure code. Before that, IaC was not really programmable, although most platforms allowed some kind of programming-like functions, at the cost of a huge learning curve that required developers to learn a proprietary solution.
That’s when the second category of IaC came up in 2019[1] when Amazon released the first version of their Cloud Development Kit (CDK). CDK basically enabled developers to create their AWS services through a variety of programming languages, such as typescript and python for example. Up to this date, Amazon AWS is the only platform that has a cloud development kit, but we believe that this is the future of IaC for other platforms as well.
Automating your infrastructure inside AWS
Currently, we can develop our IaC stacks through a service called AWS Cloudformation. The service is the base of IaC in the platform, and it allows us to define our infrastructure through objects in a YAML or JSON file.
As I said earlier, this type of definition enables some kind of programmability but does not really contain all the power and possibilities we have when using a real programming language.
If you want to integrate your IaC deployment to an existing pipeline, or simply do not want to use Cloudformation directly, you can also make use of CDK with the programming language you are most comfortable with. Currently, you can create your stack using languages such as TypeScript, JavaScript, Python, Java, and C#[2], with ongoing development to support other languages as well.
Regardless of whether you chose to develop your stack using pure Cloudformation or CDK, it is important to mention that even when using CDK, the result of your deployment will always be translated to a Cloudformation stack. Therefore, understanding the basics of Cloudformation could also be useful for CDK developers, in order to check whether your desired stack is being deployed correctly.
Programming with AWS CDK
At Zigbang, we focus mostly on developing our infrastructure code using CDK. Our code is developed mainly in TypeScript and integrated with our CI/CD pipelines. Based on our experience, we will discuss and explain the use of CDK in our deployment process.
Understanding CDK
As I said earlier, CDK is simply a development kit that allows you to define your AWS infrastructure through a programming language. The goal of this article is not exactly to give you a tutorial to build your first CDK application, but instead to give you an idea of how and why you should use CDK to escalate and automate your infrastructure.
Bellow, we have a simple CDK example using TypeScript, similar to the one we showed in our Cloudformation section, so you can compare and have an idea of how easy you can start building infrastructure with CDK.
In case you decide to try CDK yourself, please check their documentation in your desired programming language. You can also find many examples available in their GitHub repository, in case you don’t have a stack of your own.
Benefits of using CDK
- It is an open-source project, feel free to contribute!
- It allows you to version your infrastructure inside of your source code.
- It allows you to develop your infrastructure with the language you love the most, and integrate it with your existing code.
- Enables you to document your infrastructure with code. Anyone with access to the source could easily understand your infrastructure.
- It easily connects your infrastructure to your existing CI/CD pipeline.
- Allows deletion of entire environments through Cloudformation, enabling you to roll back or completely change your infrastructure with only one click.
- Enables you to reuse your infrastructure code throughout many different projects.
Cons of using CDK
- It is still in an early stage, so it could have some unknown problems when deploying some services.
- Because it is still in its early builds, the packages can changes frequently, with some breaking changes down the line. You should always check before you update your CDK version.
- The documentation is still lacking, and often times the only source of documentation is the own source code. Be prepared to do some digging from time to time. At least the project is open source so we can investigate and understand the code ourselves.
- The community support is still very low, so even when you have problems it might be hard to find other people with similar problems. Often times I research for Cloudformation instead of CDK, so I can figure out the underlying problem.
CDK-Patterns — Customization and Reusability
Deploying a single service with CDK has many advantages, but the true capacity of the development kit shines when you work with many services that use similar infrastructure. In Zigbang’s architecture, we treat each service individually, with a separate environment of its own. Being able to release new services at any point in time is the biggest reason why we at Zigbang decided to build our solutions around CDK.
We first started our CDK-Patterns project to try to abstract the issue of duplicating our infrastructure code across our projects. Our goal was to build a unique centralized library that could handle the deployment of our service with the minimum amount of work required. In order to do so, we shaped the library to represent our base service’s architecture, separating the deployment into 2 layers, the app layer, and the edge layer.
Understanding the App layer
Our app layer is the infrastructure related to the services that host our application. In our current architecture, we have the ability to deploy using both ECS and Fargate, as well as Lambda Functions. Our library allows customization of many parameters for both ECS and Lambda deployments, and also allows the use of both types of deployment at the same time. The main purpose of having a service deployed to both types of deployment is so we can handle redundancy inside of our architecture, in case Ecs or Lambda services are unavailable.
Bellow, we have the minimum amount of CDK code necessary for the client to deploy the infrastructure and host an application inside of a Lambda function.
In this article, we won’t dive too deep into all the steps necessary to build a new application, but it is also important to mention we have other required files and steps for a service to be deployed. For example, the YAML with the parameters of our function (serverless framework), and the Dockerfile used for our ECS deployment.
Understanding the Edge layer
The idea of this layer was to separate everything related to the entry points of our services into a unique optional layer. The reason why this layer is optional is because not every service actually needs an edge architecture. For example when we have internal applications with no external access. Currently, this layer is responsible for creating all the registries needed in Route53, as well as deploying and configuring the Cloudfront instance that will be in front of our application.
Bellow, we have the minimum code needed in order to deploy the edge of a simple lambda function. As you can see, both the app layer and the edge layer have very similar parameters. Because of that, in the client code, we usually have a base class to handle all the common configurations.
One size does not fit all
The fact we have a centralized library to help us to deploy our services does not necessarily mean this will fit every project and every situation. We are constantly updating our library to add more options and customization as needed. The idea is that the project can follow our base architecture and evolves together with it.
Exceptions
We are aware of the fact some services require a totally customized infrastructure. In this scenario, we still mostly use CDK to deploy our applications (as much as possible) but develop a custom stack for the service being deployed.
Conclusion
Regardless of the cons we mentioned in this article, we believe the benefits of developing with CDK outweigh every cons we faced so far. And although we didn’t go too deep into developing with CDK itself, or the source code of our library, we hope that our architecture can inspire you to look into more details, so you can start developing with CDK by yourself.
