Why we should use Terraform and Terragrunt to manage Kubernetes (with example code)
Background
Kubernetes (k8s) is gaining much popularity recently. However, the learning curve is also very large and the official documents are not doing a good job in regards to applying to the best practice in the industry environment. If you follow the official documents, you might end up with writing yaml files and using kubectl to provision these resources.
Why we should not use kubectl and yaml?
In the real world, there might be hundreds of resources and each resource might be deployed to different environments such as dev, qa, sandbox, staging, production, etc. This makes it extremely challenging for the kubectl and yaml file way to track and manage the resources changes, not to mention the overhead of DevOps engineer team changes.
How to tackle this issue?
You might heard of Infrastructure as Code (IaC), which has been widely used in managing and provisioning resources in cloud services such as AWS, GCP, Azure, etc. Luckily, we can also use IaC for k8s management. In this article, I will create a demo using Terraform and Terragrunt (a wrapper of terraform to make the IaC code dry and maintainable), which might be the most widely used IaC tools nowadays. But you can always feel free to use other tools such as Pulumi, Ansible, etc.
Benefit of using Terraform and Terragrunt
Below are some benefits of using terraform and terragrunt to manage k8s:
- IDE support - Use the same configuration language to provision the Kubernetes infrastructure and to deploy applications into it. IntelliJ Idea also provides code completion for Terraform code.
- Drift detection: Terraform plan will always present you the difference between reality at a given time and config you intend to apply.
- Full lifecycle management: Terraform doesn’t just initially create resources, but offers a single command for creation, update, and deletion of tracked resources without needing to inspect the API to identify those resources.
- Synchronous feedback: While asynchronous behaviour is often useful, sometimes it’s counter-productive as the job of identifying operation result (failures or details of created resource) is left to the user. e.g. you don’t have IP/hostname of load balancer until it has finished provisioning, hence you can’t create any DNS record pointing to it.
- Graph of relationships: Terraform understands relationships between resources which may help in scheduling — e.g. if a Persistent Volume Claim claims space from a particular Persistent Volume Terraform won’t even attempt to create the PVC if creation of the PV has failed.
Best Practice
Prerequisites
minikube: to create a local k8s cluster on your laptop https://minikube.sigs.k8s.io/docs/start
kubectl: to manage, test, and verify k8s services (We don’t need this in prod environment, only used to verify our terraform results) https://kubernetes.io/docs/tasks/tools/
tfenv: to install terraform https://github.com/tfutils/tfenv
tgenv: to install terragrunt https://github.com/cunymatthieu/tgenv
Demo code this article: https://github.com/yinchuandong/k8s-terraform-demo
Demo Code Structure
├── live # entrypoints of multiple environment
│ ├── dev
│ │ ├── env.hcl
│ │ └── k8s
│ │ ├── terraform.tfstate # auto-generated (only available using local state)
│ │ ├── terraform.tfstate.backup # auto-generated (only available using local state)
│ │ └── terragrunt.hcl
│ ├── prod
│ │ ├── env.hcl
│ │ └── k8s
│ │ ├── terraform.tfstate # auto-generated (only available using local state)
│ │ ├── terraform.tfstate.backup # auto generated (only available using local state)
│ │ └── terragrunt.hcl
│ └── terragrunt.hcl # main entrypoint
└── modules # shared resource
└── k8s
├── context.tf
├── main.tf
├── output.tf
└── variables.tfShared modules:
Live (entry points of multiple stages):
Tutorial (with code)
Start minikube cluster
Start minikube mangement web console UI
Open the URL in browser and you shall be able to see the UI below
Go to live folder and run the following commands subsequently to setup correct terraform and terragrunt version:
- tfenv install <your version>
- tfenv use <your version>
- tgenv install <your version>
- tgenv use <your version>
Go to live/dev/k8s, run:
- terragrunt init
- terragrunt validate
Then run terragrunt plan and you shall be able to see the resources to be added, updated, or deleted. Note that this operation will not change any resources, but just tells you what will be changed.
Then run terragrunt apply to apply the changes if you feel happy with the above changes. Note that this command will ask your confirmation, you will can disable the confirmation in CI otherwise it will be paused forever.
You shall be able to see the resource in minikube web console UI if it is applied successfully.
You can also access the demo nginx website in browser via http://<your own k8s ip>:<exposed port> . Note that you can use minikube ip to get the external IP address.
You can do the same step to provision production environment in live/prod/k8s.
Summary
At the end of the article, you should be able to successfully provision two k8s environments (dev and prod) via Terraform and understand the benefits. Note that this article only shows a very simple demo, but it presents a clean and dry code structure for you to refer to. Feel free to check the official terraform documents and dive into it!
