CDK or Terraform?

CDK vs Terraform

I know, I know it is late. Picking an infrastructure provider in 2020, what had I been doing all that time… That’s true, but hey at least we are having the final say — who do we want to use for delivering serverless products to the cloud?

Decisions, decisions

Recently, I had to work on a streak of serverless products. It was a blast, a lot of decisions were made, I learned a lot, I taught a lot. All good, all that Arnold taught us (better yourself, give back, and so on).

However, one decision bugged me for some time — what is the best tool to deploy resources to the cloud? This is no small matter — in serverless provisioning is important. Sometimes it is even harder than making the resources (the code) themselves. Let’s do it right.

What do we have, what do we need

So, the Company is “Serverless first” and we like AWS. That’s the first constraint — we only need to be able to deploy AWS resources. The second constraint — the infrastructure-as-a-code approach must be convenient for developers. We don’t have a separate team of people who would be experts in just that — in the Company, a single person develops code for a lambda function and for the deployment script of that lambda. Then, the deployment framework must be mature, must be supported. We are not a startup, we want stability and features even by the price of doing things the old-fashioned way.

So, this is it — AWS, comfortable to write, mature. Truth be told, I did not do vast research, but here are the options that I considered:

• Terraform
• AWS CDK
• Pulumi

I discarded Serverless, Cloud Formation, AWS SAM as either not mature enough (Serverless, like honestly, if you can find something more than a Lambda+API Gateway it would be insane), or not convenient for developers (nobody wants to write miles of Yaml in Cloud Formation).

Among the three, Pulumi failed the hello-world test. I tried to run a simple Golang lambda example from their docs and it failed with an error. Please hear me out — I did not do any effort on fixing the error or even understanding it. For me, it was enough to see that in the conditions I have the simplest example did not work. The example from the docs. If we fail on step one — let’s not even bother.

Please, take my Pulumi evaluation with a fine deal of skepticism — it could totally work for you and by many is believed as the best tool for the task. I just felt discouraged in the first 15 minutes I used it and just discontinued it.

So then we are left with Terraform and AWS CDK. Let’s take a deeper look.

AWS CDK

Created by Amazon themselves, a new product. It employs the infrastructure-as-a-code to the fullest — you can write your infrastructure code in the language of your choice. Typically, we use TypeScript. Underneath, AWS CDK converts the code into a Cloud Formation template and the provision then delivered by it.

Terraform

Created by HashiCorp, it has been around for a while. Supports multiple cloud providers including AWS. Terraform takes more of a declarative approach, resources are described using HCL (HashiCorp Configuration Language) — a configuration language that is supposed to be both human and machine friendly. The language is kind of like YAML. It is not YAML, but it is just to give you an idea of what to compare it with.

Let’s try to put the popularity and the release date for the comparison:

Note that I included Terraform and Terraform AWS Module separately, as it is more representative of my case. I only care about the AWS part, so even though Terraform was released in 2014 and CDK only in 2018 it doesn’t mean that Terraform’s AWS support is three times more mature that CDK’s. If we compare the Terraform AWS Module and AWS CDK the difference is not that big — just one year. But a year is a year, you know. I think it is still a fairly big difference.
The same goes for popularity in the community. While Terraform itself has about 24k stars, the Terraform AWS Module and AWS CDK have roughly the same love. Having said that, I believe that there are quite a lot of people who would just never visit the Terraform AWS Module’s repo and simply star what they can Google by “terraform github”. This means that Terraform’s 24k stars include those who are “AWS Only” and this must be considered.

To put it together kudos to Terraform for long life and much love. Terraform wins the maturity award in my eyes.

You might say — “Hey, but what about the fact that AWS CDK is written by Amazon, does it not make it more stable by definition? These people, they know their stuff”. I agree with that, however, it is impossible to measure, so as soon as there is no strong evidence for that, I will discard this. I believe time is one of the most critical factors when it comes to maturity.

Now, to the best part — how fun is it to write code in both?

The actual fun, seriously

For the poor souls coming from the Java world the best explanation I can give when comparing CDK and Terraform — it is exactly like Gradle and Maven. The first one is code, the second one is a declaration. CDK gives you the freedom to do whatever, Terraform is more conservative — you need to write what is within the scope. I am not going to go through all the Declarative vs Imperative again. If you are from the Java world, you most likely know already. CDK will give you flexibility and convenience in tough situations, but reading unfamiliar CDK code will surprise you. Terraform is more organized, but this is limiting.

Where CDK is winning is that for your project you can choose from a list of programming languages it supports. This is immensely cool, when everything is written (code + infra) in a single language — it is big. It is not as big as some people might think of, as in practice your infra code even if written in TypeScript will never be using any complex language features. You will never integrate your infra code with source code. Esthetically you will be pleased though.

For those who don’t understand what is it all about — just ignore it, we will just go through some practical example. I believe in practice, so yeah, let’s practice.

Into the Real world

From now on I will be referring to a piece of infra code (stripped and anonymized) that I have written in both Terraform and CDK. The repository can be found here: https://github.com/Otanikotani/tf-vs-cdk. It is not important what exactly this code deploys. It is just some AWS Glue jobs and triggers, S3 buckets (empty and with assets), and AWS Neptune cluster. There are also some lookups (existing resources not declared in the code) for networking resources, variables, and dependencies between different resources. This is a solid example, not a “hello world”. Let’s go through some concepts and try to evaluate both frameworks accordingly.

Structure

For the CDK we used TypeScript so that modular structure is inherited. There are certain best practices in CDK. Have a bin/<name-your-app>.ts file and lib/ folder contains all the stacks. A stack is a single module of your app. You can always choose to deploy a single stack if you like (not sure why though, but yeah). There are some hard limits on the size of the stack — it can’t have more than 100 resources, but it is unlikely to be hit in a well-designed infra code. Not a factor. In the sample repository, we have three stacks: Neptune, Network, and Orchestration.

Stacks can be dependent on each other — since Neptune requires Network resources, you can see that the Network stack is passed as an argument to Neptune. This is very handy and comfortable. Just like in normal programming. I once ran into an issue of circular dependencies in CDK and apparently, it is quite easy to run into it. CDK couldn’t resolve it properly even though I think I did not have any cycles in the dependency graph of the stacks. Oh well, just simplify things then, not too much of a trouble.

In Terraform there are Modules. Each module can contain multiple .tf files. So in tf/ you will find a main.tf and three subfolders named after the modules. The modules are identical to what we have in CDK. The difference here is building the dependencies.

While in CDK it was easy, Terraform requires you to jump through. For example, to pass a Neptune endpoint URL from the AWS Neptune module to Orchestration I had to:

• Declare a variable variable “neptune_endpoint” { } in the Orchestration module.
• In root main.tf I had to set the value for that variable neptune_endpoint = module.neptune.neptune_cluster_endpoint

This is much less convenient for me. I would prefer something simpler, CDK’s simplicity is more likable.

CDK wins the Structure by both predictability and simplicity.

Writing

I used IntelliJ IDEA Ultimate with Terraform plugin version 0.7.10 for this article. I did not use anything special for CDK — just your normal TypeScript. What I want to compare here is — how comfortable was it to write in HCL (Terraform’s language) and CDK. I am no expert in both, so it is a fair comparison.

In Terraform, I loved that once I declared that I will be using AWS provider the autocomplete started to work flawlessly and everywhere. I can type resource “aws_ and then all the options will be there. This is very easy, this reduces Googling for names of certain modules/resources a lot. What annoyed me though is that whenever you autocomplete a variable in Terraform it will automatically add a double quote:

Terraform autocomplete

This is extremely annoying for cases when you don’t need the quotes when you want to reference another resource.

CDK support was nothing special — it is a typical TypeScript. What I did not like there (in comparison with Terraform) is that I had to add a module one by one to the package.json to add new resources. I like having everything together and just type until I narrow the autocomplete search to what I need.

Terraform wins the writing experience but by a tiny margin.

Verbosity

Compare the following two pieces of code:

CDK:

Terraform:

While in most cases the code in TypeScript and HCL look identical in size here you can sense the difference. These two fragments of code create the same thing in AWS — an IAM role. I hated it when I wrote a resource after resource in Terraform. CDK’s declaration looks cute and concise, Terraform — I could’ve made a mistake in so many places there, it is just too much.

CDK is less verbose. Especially if you use functions, interfaces, etc available in TypeScript.

Performance

I would say that infrastructure code performance is secondary, but still. If it is a part of the daily life of a developer, it is considerable. The difference between 5 and 15 minutes is significant.

Terraform deploys resources using AWS SDK. CDK first converts the resources to Cloud Formation templates and then applies the template.

Terraform seemed to work faster than CDK, exactly because of the Cloud Formation template conversion. For some reason, it takes a lot of time to apply a Cloud Formation template. So, Terraform wins here.

The unknown

A big part of the development is handling errors. When it comes to infrastructure code it is as important — the better Google works for errors spit out by your infrastructure framework the easier it is to figure out what went wrong. You won’t be able to debug it in a conventional way like normal code anyways.

So, here I am confused. Terraform looked (it is more mature after all) much better in the descriptive errors department. CDK sometimes gives you a run for your money to try and understand the error. And when you finally find the GitHub issue that describes the root of your error, you will notice that the fix for that issue is not yet ready. Well, Terraform is more mature after all, people used it more, reported more bugs, more bugs were fixed. CDK is only on its way there.

That being said, I am confused. When I was deploying a Neptune cluster using Terraform I ran into an issue with gRPC connection which I was not able to solve. 5 minutes in Google didn’t help also. That was baffling, it is like you are looking at a great painting and then you notice a little piece of dirt on it. Irritating.

I still think Terraform is better at explaining what went wrong when it goes wrong, so I will give it the victory, but my eyes are on you, Terraform! Don’t do this gRPC to me again.

And the Winner is…

I will go with CDK. Not only having a single language for the whole product is pleasing, but it also seems that you need to write less to achieve more. That’s important to me.

The choice is yours though, if the advantages of Terraform (and there are many) are what you need — go for it!