Terraform Kubernetes Integration with Minikube
Happy devSecOps
Infrastructure as Code(IaC)
Infrastructure-as-Code(IaC) is a practice of managing and provisioning of infrastructure through code instead of through manual processes. This infrastructure mostly involves cloud infrastructure and a bit of on-prem infrastructure. For an example hardware, software, Operating Systems and data storage. IaC allows for the definition, version control, and automation of infrastructure components and configurations. This means that the same processes and techniques used in software development can be applied to IT infrastructure management, making it more efficient, repeatable, and scalable. IaC helps in reducing manual errors, increasing collaboration among teams, and making infrastructure changes more transparent and easier to rollback if necessary. Popular tools for IaC include Terraform, Puppet, Chef, and Ansible.
About Terraform
Terraform is open-source IaC tool created by Hashicorp. It allows users to define, configure, and manage their infrastructure resources in a declarative manner, using code written in the Terraform language. The code can be version controlled, making it easy to track changes to the infrastructure and roll back to a previous state if necessary.
Terraform is mostly used to provision cloud infrastructure as it supports a lot of providers such as AWS, Azure, GCP, Alibaba Cloud, Oracle Cloud. Terraform does support on-prem infrastructure but not stand-alone infrastructure, the on-prem infrastructure must either be operating with OpenStack or VMWare as these are supported by Terraform. This makes Terraform a versatile tool for managing and automating infrastructure in a multi-cloud or hybrid cloud environment.
Following figure shows the architecture of Terraform. It contains 4 main components, 1) Terraform config files(.tf), 2) Terraform state file(.tfstate), 3)Terraform core, 4) Providers. Terraform config files contain the code written in the Terraform language that defines the infrastructure to be managed. Terraform uses the configuration files to create and manage resources in the target infrastructure. Terraform maintains a state file that records the state of the infrastructure being managed. This file is used by Terraform to keep track of changes and ensure that the infrastructure remains in a desired state. Terraform Core is the central component of the Terraform tool and includes the Terraform CLI. The source code of the core(which written with golang) is open source and hosted github.com/hashicorp/Terraform. The Terraform config files are analysed by the core and compared with Terraform state file(which holds the current state of the infrastructure) in order to decide on the point of action which is either to create, update or destroy infrastructure. Terraform relies on plugins called providers to interact with cloud providers, SaaS providers, and other APIs. There are various Terraform provider plugins available aws, github, gcp, azure, kubernets to handle different infrastructures. Once the state of the infrastructure has been determined via the Terraform config file, the Core then interacts with the desired cloud provider via Terraform plugin(through RPC commands) to create, update or destroy the desired infrastructure. Terraform Core is the central component of the Terraform tool. The Terraform CLI provides a command line interface for executing Terraform commands(e.g init, plan, apply) and is the primary way that users interact with Terraform Core. The CLI uses the Terraform configuration files and state files to manage and provision resources in the target infrastructure.
Terraform Kubernets Provider
The Terraform Kubernetes provider is a plugin for Terraform that allows users to manage Kubernetes resources using Terraform. It enables Terraform to interact with the Kubernetes API to create, update, and delete resources in a Kubernetes cluster.
With the Terraform Kubernetes provider, users can define their Kubernetes resources in Terraform configuration files and manage them in a declarative manner, just like other infrastructure resources. This makes it possible to use Terraform to manage the complete lifecycle of a Kubernetes application, from initial deployment to ongoing maintenance.
The Terraform Kubernetes provider supports a wide range of Kubernetes resources, including pods, services, and deployment configurations. It also supports Kubernetes extensions, such as Custom Resource Definitions (CRDs), and can be used to manage both on-premises and cloud-based Kubernetes clusters.
In this post I’m gonna discuss about using Terraform Kubernets provider to manage Kubernets resources on Minikube based kubernets cluster. All the sources codes which related to this post available in gitlab. Please clone the repo and continue the post.
Install Terraform
I have installed Terraform using asdf(all-in-one version manager) which is a tool that allows developers to manage multiple versions of programming languages and tools on the same machine. Following is the way to install terraform via asdf.
# install asdf
❯❯ brew install asdf
# add asdf terraform plugin
❯❯ asdf plugin-add terraform
# install terraform latest version via asdf
❯❯ asdf install terraform latest
# install specific version of terraform via asdf
❯❯ asdf install terraform 1.1.5
# set global terraform version
❯❯ asdf global terraform 1.1.5
# specify local version of terraform in a file called .tool-versions
❯❯ cat .tool-versions
terraform 1.1.5Kubernets Cluster Setup
I have configured Kubernets cluster in local environment using Minikube. I have used Minikube to run a Kubernetes cluster on local environment. But it also runs a Docker daemon that can be used to run containers. Hyperkit is an open-source hypervisor for macOS hypervisor, optimized for lightweight virtual machines and container deployment. Read more about Minikube setup from here.
Terraform Kubernets Provider Setup
The Terrform Kubernetes provider is used to interact with the resources supported by Kubernetes. To interact with Kubernetes cluster via Terraform first I have to define and configure the Terraform Kubernetes provider. Following is the way to define and configure Kubernets provider in the provider.tf file. The syntax of the file is HashiCorp Configuration Language(HCL) which is a unique configuration language. In the provider.tf I have defined the Kubernets plugin in required_providers and specified the Kubernets configurations(e.g config file path, context). Read more about Kubernets provider configuration from here.
terraform {
required_providers {
kubernetes = {
source = "hashicorp/kubernetes"
version = "2.17.0"
}
}
}
provider "kubernetes" {
config_path = "~/.kube/config"
config_context = "minikube"
}Then need to initialize the provider via terraform init. It will download the specified version of the Kubernetes provider into the directory called .terraform.
# init provider
❯❯ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/kubernetes versions matching "2.17.0"...
- Installing hashicorp/kubernetes v2.17.0...
- Installed hashicorp/kubernetes v2.17.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
# it will download kubernets provider into .terraform directory
❯❯ la -l .terraform
total 0
drwxr-xr-x 3 eranga staff 96B Feb 4 18:56 providers
❯❯ ls -l .terraform/providers/registry.terraform.io/hashicorp/kubernetes
total 0
drwxr-xr-x 3 eranga staff 96 Feb 4 18:56 2.17.0Create Kubernets Namespace with Terraform
Once the Kubernetes provider is configured, I can define Kubernets resources in Kubernets config file. Following is the way to define Terraform resource which related to Kubernets namespace. I have defined these configuration in a file named main.tf.
resource "kubernetes_namespace" "rahasak" {
metadata {
name = "rahasak"
}
}Once defined the Terraform configurations, I can validate the syntax of the config file via terraform validate and preview the changes that will be made to the infrastructure(e.g to the Kubernets cluster) before actually applying them via terraform plan. The terraform plan command should be run before apply the changes to the infrastructure. This allows you to preview the changes and make any necessary modifications before actually applying them. Following is the output, it will shows what are the that will be made to the Kubernets cluster. In this case creating a namespace classed rahasak.
❯❯ terraform validate
Success! The configuration is valid.
❯❯ terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# kubernetes_namespace.rahasak will be created
+ resource "kubernetes_namespace" "rahasak" {
+ id = (known after apply)
+ metadata {
+ generation = (known after apply)
+ name = "rahasak"
+ resource_version = (known after apply)
+ uid = (known after apply)
}
}
Plan: 1 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
# if want we can save terraform plan output in to file and read it later
# here we create a text file with ANSI control characters
❯❯ terraform plan -out tf.plan
❯❯ terraform show tf.plan > tfplan.ansi
# view plan output
❯❯ less -R tfplan.ansiNow I can apply the changes that defined in the Terraform config file(main.tf) to the Kubernets cluster via terraform apply command. Basically, terraform apply is used to apply changes to the infrastructure that are defined in the Terraform configuration files. The terraform apply command takes the execution plan generated by the terraform plan command and makes the actual changes to the infrastructure. Following is the output of terraform apply. It will create a namespace called rahasak in the Kubernets cluster.
❯❯ terraform apply
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# kubernetes_namespace.rahasak will be created
+ resource "kubernetes_namespace" "rahasak" {
+ id = (known after apply)
+ metadata {
+ generation = (known after apply)
+ name = "rahasak"
+ resource_version = (known after apply)
+ uid = (known after apply)
}
}
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
kubernetes_namespace.rahasak: Creating...
kubernetes_namespace.rahasak: Creation complete after 0s [id=rahasak]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
# rahasak namespace created
❯❯ kubectl get ns
NAME STATUS AGE
default Active 6d4h
kube-node-lease Active 6d4h
kube-public Active 6d4h
kube-system Active 6d4h
rahasak Active 22sThe terraform apply command generates Terraform state file called terraform.tfstate. This state file is stored by default in a local filesystem, but it can also be stored remotely(e.g AWS S3 bucket etc). When run the terraform apply command, Terraform will take the execution plan generated by the terraform plan command and make the changes to the infrastructure. As Terraform makes these changes, it updates the state file with the new state of the infrastructure. This includes information about the resources that were created, updated, or deleted, as well as their properties and relationships with other resources. The state file is updated every time we run the terraform apply command. When running the terraform plan, this state file is used determine the changes that need to be made to the infrastructure. Basically, Terraform compares the information in the state file with the information in the Terraform configuration files to generate an execution plan. The execution plan outlines the changes that will be made to the infrastructure when we run the terraform apply command. Following is the content of terraform.tfstate file.
❯❯ cat terraform.tfstate
{
"version": 4,
"terraform_version": "1.1.5",
"serial": 3,
"lineage": "564e0598-11cb-c83d-fa93-7f859d4c201f",
"outputs": {},
"resources": [
{
"mode": "managed",
"type": "kubernetes_namespace",
"name": "rahasak",
"provider": "provider[\"registry.terraform.io/hashicorp/kubernetes\"]",
"instances": [
{
"schema_version": 0,
"attributes": {
"id": "rahasak",
"metadata": [
{
"annotations": null,
"generate_name": "",
"generation": 0,
"labels": null,
"name": "rahasak",
"resource_version": "260720",
"uid": "87d39cd8-a8ea-45c3-b6a9-445f8a6db7b0"
}
],
"timeouts": null
},
"sensitive_attributes": [],
"private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiZGVsZXRlIjozMDAwMDAwMDAwMDB9fQ=="
}
]
}
]
}Create Nginx Deployment with Terraform
Following is the Terraform configuration to create Nginx deployment. This configuration will create a scalable Nginx Deployment with 2 replicas. It will expose the Nginx frontend using a Kuberntes service of type ClusterIP.
resource "kubernetes_deployment" "nginx" {
metadata {
name = "nginx"
namespace = kubernetes_namespace.rahasak.metadata.0.name
}
spec {
replicas = 2
selector {
match_labels = {
app = "nginx"
}
}
template {
metadata {
labels = {
app = "nginx"
}
}
spec {
container {
image = "nginx"
name = "nginx"
port {
container_port = 80
}
}
}
}
}
}
resource "kubernetes_service" "nginx" {
metadata {
name = "nginx"
namespace = kubernetes_namespace.rahasak.metadata.0.name
}
spec {
selector = {
app = kubernetes_deployment.nginx.spec.0.template.0.metadata.0.labels.app
}
port {
port = 80
}
}
}Now when I’m executing terraform plan, it will compares the state with terrafrom.tfstate file and generates generate an execution plan. Execution plan contains the information about Nginx deployment and service.
❯❯ terraform plan
Terraform will perform the following actions:
# kubernetes_deployment.nginx will be created
# kubernetes_service.nginx will be created
Plan: 2 to add, 0 to change, 0 to destroy.Then I can apply the new changes that defined in the Terraform config file(main.tf) to the Kubernets cluster via terraform apply command. It will create Nginx deployment and ClusterIP service for Nginx frontend.
❯❯ terraform apply
Plan: 2 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
kubernetes_deployment.nginx: Creating...
kubernetes_deployment.nginx: Still creating... [10s elapsed]
kubernetes_deployment.nginx: Still creating... [20s elapsed]
kubernetes_deployment.nginx: Creation complete after 26s [id=rahasak/nginx]
kubernetes_service.nginx: Creating...
kubernetes_service.nginx: Creation complete after 0s [id=rahasak/nginx]
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
❯❯ kubectl get pods -n rahasak
NAME READY STATUS RESTARTS AGE
nginx-54b5bd6994-5jfbn 1/1 Running 0 2m5s
nginx-54b5bd6994-czs4j 1/1 Running 0 2m5s
❯❯ kubectl get svc -n rahasak
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx ClusterIP 10.104.68.182 <none> 80/TCP 104s
❯❯ kubectl port-forward svc/nginx 8080:80 -n rahasak
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
❯❯ curl localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>Manage Terraform State
The state file is updated every time we run the terraform apply command. When running the terraform plan, this state file is used determine the changes that need to be made to the infrastructure. Terraform provides commands to manage the current state.
# list states
❯❯ terraform state list
kubernetes_deployment.nginx
kubernetes_namespace.rahasak
kubernetes_service.nginx
# get state
❯❯ terraform state show kubernetes_service.nginx
# kubernetes_service.nginx:
resource "kubernetes_service" "nginx" {
id = "rahasak/nginx"
status = [
{
load_balancer = [
{
ingress = []
},
]
},
]
wait_for_load_balancer = true
metadata {
generation = 0
name = "nginx"
namespace = "rahasak"
resource_version = "261880"
uid = "d0ef1ba0-f599-4cc4-b9ff-ff8e19c6622c"
}
spec {
allocate_load_balancer_node_ports = true
cluster_ip = "10.104.68.182"
cluster_ips = [
"10.104.68.182",
]
health_check_node_port = 0
internal_traffic_policy = "Cluster"
ip_families = [
"IPv4",
]
ip_family_policy = "SingleStack"
publish_not_ready_addresses = false
selector = {
"app" = "nginx"
}
session_affinity = "None"
type = "ClusterIP"
port {
node_port = 0
port = 80
protocol = "TCP"
target_port = "80"
}
}
}
# remove state
# once delete the state terraform plan will shows that, it need to add deleted resource
❯❯ terraform state rm kubernetes_service.nginx
Removed kubernetes_service.nginx
Successfully removed 1 resource instance(s).
# delete all state objects
❯❯ terraform state rm $(terraform state list)
Removed kubernetes_deployment.nginx
Removed kubernetes_namespace.rahasak
Successfully removed 2 resource instance(s).As shown above we can delete the Terraform state from the .tfstate file. Deleting state does not mean that it deleting the actual resources(e.g deleting Kubernets pods, services etc). Once delete the state, terraform plan will shows that, it need to add deleted resource. For an example in above scenario when deleting the kubernetes_service.nginx terraform plan will shows that it need to create kubernetes_service.nginx resource. But when we run terraform apply, it will fail since kubernetes_service.nginx already existing and running. We need to take existing infrastructure components and import them into Terraform state. Terraform provide command terraform import for that.
# run terraform apply after deleting state objects
# it will fail since the resource already existing
# we need to import the existing resouces back to terraform state
❯❯ terraform apply
Plan: 3 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
kubernetes_namespace.rahasak: Creating...
╷
│ Error: namespaces "rahasak" already exists
│
│ with kubernetes_namespace.rahasak,
│ on main.tf line 1, in resource "kubernetes_namespace" "rahasak":
│ 1: resource "kubernetes_namespace" "rahasak" {
│
# import state object of namespace
❯❯ terraform import kubernetes_namespace.rahasak rahasak
kubernetes_namespace.rahasak: Importing from ID "rahasak"...
kubernetes_namespace.rahasak: Import prepared!
Prepared kubernetes_namespace for import
kubernetes_namespace.rahasak: Refreshing state... [id=rahasak]
Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
# import state object of nginx deployment
❯❯ terraform import kubernetes_deployment.nginx rahasak/nginx
kubernetes_deployment.nginx: Importing from ID "rahasak/nginx"...
kubernetes_deployment.nginx: Import prepared!
Prepared kubernetes_deployment for import
kubernetes_deployment.nginx: Refreshing state... [id=rahasak/nginx]
Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
# import state object of nginx service
❯❯ terraform import kubernetes_service.nginx rahasak/nginx
kubernetes_service.nginx: Importing from ID "rahasak/nginx"...
kubernetes_service.nginx: Import prepared!
Prepared kubernetes_service for import
kubernetes_service.nginx: Refreshing state... [id=rahasak/nginx]
Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.Delete Terraform Resources
Terraform providers terraform destroy command for removing Terraform-managed infrastructure components/resources. The terraform destroy command is used to remove resources that were created by Terraform and update the state file to reflect the changes. Following is the way to delete the resources with terraform destroy command.
# destroy single resource
❯❯ terraform destroy -target=kubernetes_service.nginx
Plan: 0 to add, 0 to change, 1 to destroy.
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.
│
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests
│ to use it as part of an error message.
╵
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
kubernetes_service.nginx: Destroying... [id=rahasak/nginx]
kubernetes_service.nginx: Destruction complete after 0s
╷
│ Warning: Applied changes may be incomplete
│
│ The plan was created with the -target option in effect, so some changes requested in the configuration may have been ignored and the output values may not be fully updated.
│ Run the following command to verify that no other changes are pending:
│ terraform plan
│
│ Note that the -target option is not suitable for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform
│ specifically suggests to use it as part of an error message.
╵
Destroy complete! Resources: 1 destroyed.
# destroy all resource
❯❯ terraform destroy
Plan: 0 to add, 0 to change, 2 to destroy.
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
kubernetes_deployment.nginx: Destroying... [id=rahasak/nginx]
kubernetes_deployment.nginx: Destruction complete after 0s
kubernetes_namespace.rahasak: Destroying... [id=rahasak]
kubernetes_namespace.rahasak: Destruction complete after 6s
Destroy complete! Resources: 2 destroyed.Reference
- https://opensource.com/article/20/7/terraform-kubernetes
- https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/guides/getting-started
- https://dev.to/chefgs/deploy-kubernetes-resources-in-minikube-cluster-using-terraform-1p8o
- https://devpress.csdn.net/opensource/62f536a17e6682346618a54e.html
- https://www.hashicorp.com/blog/managing-google-calendar-with-terraform
