Maximizing Cost-Efficiency with Karpenter : Setting Up EKS Cluster (v1.24) on Spot-Instances with AWS ALB Ingress Controller, Amazon EBS CSI Driver, and Add-ons Completely using Terraform and Terraform Cloud
In this blog you will learn how to provision EKS Kubernetes cluster at any scale using Karpenter . We will also utilize spot instances to reduce costs by up to 90 percent. Additionally, we’ll set up the AWS ALB Ingress Controller and Amazon EBS CSI driver, establish trust between an OIDC-compatible identity provider and your AWS account using IAM OIDC identity provider (including the necessary IAM roles and policies), install the metrics server, and set up required EKS add-ons using Terraform. We will deploy all of this using Helm charts and configure custom values.yml using Terraform as well. Finally, we will use Terraform Cloud for remote state, creating workspaces, and the plan/apply workflow.
If you would like to skip the theoretical explanations please fell free scroll down to the actual terraform configurations and where do the testings!!
Kuberenetes Autoscaling
A Kubernetes Cluster is a group of node machines that run containerized applications. Inside these nodes, Pods run containers that demand resources such as CPU, memory, and sometimes disk or GPU.
Getting the size of a Kubernetes cluster right is not an easy task, if the number of nodes provisioned is too high, resources might be underutilized and if it’s too low, new workloads won’t be able to be scheduled in the cluster.
Setting the number of nodes manually is a simple approach, but it requires manual intervention every time the cluster needs to grow or to shrink, and it will make nearly impossible to adapt the cluster size to cover rapid traffic and load fluctuations.
One of the benefits of using Kubernetes is its ability to dynamically scale your infrastructure based on user demand.
Kubernetes offers multiple layers of auto-scaling functionality, including:
Pod-based autoscaling
- Horizontal Pod Autoscaler — adds or removes more pods to the deployment as needed
- Vertical Pod Autoscaler — resizes pod’s CPU and memory requests and limits to match the load
Node-based autoscaling: adding or removing nodes as needed
How we can reduce cost with Using Spot Instances?
AWS offers a cost-saving option for EC2 instances called “Spot instances” where AWS provides the unused/unoccupied EC2 instances for up to a 90% cheaper than regular On-Demand instances. However, their availability is not guaranteed as they are unused instances that are made available to users.
When a Spot instance is reallocated, AWS automatically provides a new instance to take its place. While Spot instances are similar to regular On-Demand instances, they come with a risk of potential interruption, but they can be an effective way to save costs for workloads that are not time-sensitive.
Spot instances are compatible with Amazon Elastic Kubernetes Service (EKS), and they can be used to scale production applications when there is a surge in demand. By leveraging Spot instances, users can take advantage of cost savings while also ensuring that their applications can handle spikes in traffic without compromising performance.
One downside of using Spot Instances is that they can be interrupted by the AWS EC2 Spot service, which is why it’s crucial to design your application with fault-tolerance in mind. To help with this, you can utilize Spot Instance interruption notices, which provide a two-minute warning before Amazon EC2 stops or terminates your instance. Keep in mind that after this notification, the instance will be reclaimed. By designing your application to handle interruptions gracefully, you can minimize the impact of Spot Instance interruptions and still benefit from the cost savings they offer.
Why use Karpenter instead of Cluster Autoscaler?
Cluster Autoscaler is a useful Kubernetes tool that can adjust the size of a Kubernetes cluster by adding or removing nodes, based on the utilization metrics of nodes and the presence of pending pods. However, it requires nodes with the same capacity to function correctly.
To adjust the capacity, Kubernetes Cluster Autoscaler interacts with the Autoscaling group service directly. To properly work , AWS EKS managed node groups are needed, and the Autoscaler only scales up or down the managed node groups through Amazon EC2 Auto Scaling Groups. Whenever a new node group is added, it is essential to inform Cluster Autoscaler about it, as there is a mapping between Cluster Autoscaler, which is Kubernetes native, and the node group, which is AWS native.
Cluster Autoscaler does not provide flexibility to handle hundreds of instance types, zones, and purchase options.
Unlike Autoscaler, Karpenter doesn’t rely on node groups ,it manages each instance directly. Instead, autoscaling configurations are specified in provisioners, which can be seen as a more customizable alternative to EKS-managed node groups.
- If you are uncertain about the instance types you require or have no specific requirements, Karpenter can make the decision for you. All you need to do is create a provisioner that outlines the minimum parameter requirements.
- Karpenter is designed to make efficient use of the full range of instance types available through AWS. Unlike other autoscaling tools that may be limited in the instance types they can use, Karpenter can select and utilize any instance type that meets the needs of incoming pods.
- Karpenter looks at the workload (i.e pods) and launches the right instances for the situation.
- Karpenter manages instances directly, without the need for additional orchestration mechanisms such as node groups. This allows it to retry capacity requests in a matter of milliseconds, significantly reducing the time it takes to address capacity issues.Moreover, Karpenter’s direct management approach enables it to make efficient use of diverse instance types, availability zones, and purchase options, without the need to create numerous node groups. This results in a more flexible and cost-effective solution for managing Kubernetes clusters on AWS.
- Infrastructure costs are reduced by looking for under-utilized nodes and removing them, also by replacing expensive nodes with cheaper alternatives, and by consolidating workloads onto more efficient compute resources.
- Karpenter operates on an intent-based model when making instance selection decisions. This model takes into account the specific resource requests and scheduling requirements of incoming pods. By doing so, Karpenter can identify the most suitable instance type to handle these pods, which may involve requesting a larger instance type capable of accommodating all the pods on a single node.
For instance, if there are 50 pending pods waiting to be scheduled, Karpenter will calculate the resource requirements of these pods and select an appropriate instance type to accommodate them. Unlike Cluster Autoscaler, which may need to scale several nodes to meet the demand, Karpenter can request a single large EC2 instance and place all the pods on it.
How it works?
Karpenter is designed to observe events within the Kubernetes cluster and send commands to the underlying cloud provider’s compute service. Upon installation, Karpenter begins to monitor the specifications of unschedulable Pods and calculates the aggregate resource requests. Based on this information, it can make decisions to launch and terminate nodes in order to reduce scheduling latencies and infrastructure costs.
To achieve this, Karpenter leverages a Custom Resource Definition (CRD) called Provisioner. The Provisioner specifies the node provisioning configuration, including the instance size/type, topology (such as availability zone), architecture (e.g. arm64, amd64), and lifecycle type (such as spot, on-demand, preemptible).
Karpenter is designed to automatically manage the lifecycle of nodes provisioned by Kubernetes. When a node is no longer needed or has exceeded its TTL (time-to-live), Karpenter triggers a finalization process that includes several steps to gracefully decommission the node.
One event that can trigger finalization is the expiration of a node’s configuration, as defined by the ttlSecondsUntilExpired parameter. This parameter specifies the maximum amount of time that a node can remain active before being terminated. When the node's TTL expires, Karpenter takes action by cordoning off the node, which means that no new workloads can be scheduled on it. Karpenter then begins to drain the pods running on the node, which involves moving the workloads to other nodes in the cluster. Finally, Karpenter terminates the underlying compute resource associated with the node and deletes the node object.
Another event that can trigger finalization is when the last workload running on a Karpenter provisioned node is terminated. In this case, Karpenter follows the same process of cordoning the node, draining the pods, terminating the underlying resource, and deleting the node object.
By automating the finalization process, Karpenter ensures that nodes are only active when they are needed, which can help to optimize the performance and cost-effectiveness of Kubernetes clusters on AWS.
How Karpenter handles Spot Interruptions
Karpenter requires a queue to exist that receives event messages from EC2 and health services in order to handle interruption messages properly for nodes
Karpenter will keep watching for upcoming involuntary interruption events that would cause disruption to your workloads.
These events include Spot Interruption Warnings, Scheduled Change Health Events, Instance Terminating Events, and Instance Stopping Events.
When Karpenter detects that one of these events is imminent, it automatically performs cordoning, draining, and termination of the affected node(s) to allow enough time for cleanup of workloads prior to compute disruption. This is particularly useful in scenarios where the terminationGracePeriod for workloads is long or where cleanup is critical, as it ensures that there is enough time to gracefully clean up pods.
Karpenter achieves this by monitoring an SQS queue that receives critical events from AWS services that could potentially affect nodes. However, to use this feature, it is necessary to provision an SQS queue and add EventBridge rules and targets that forward interruption events from AWS services to the SQS queue.
Now lets get into action !!!
I am assuming that the underlying VPC and network resources are already created.
I will be creating a seperate blog on setting up Production Grade VPC (HA) with environment-specific run method which we used .
The public subnets in the VPC must be tagged with :
"kubernetes.io/cluster/<cluster-name>" = "owned"
"karpenter.sh/discovery" = <cluster-name>
"kubernetes.io/role/elb" = 1The private subnets in the VPC must be tagged with :
"kubernetes.io/cluster/<cluster-name>" = "owned"
"karpenter.sh/discovery" = <cluster-name>
"kubernetes.io/role/internal-elb" = 1Also we will be adding the following tags to the security group of control plane and eks-nodes:
"karpenter.sh/discovery" = <cluster-name>We will be discussing this when we move ahead!!!
Creating the EC2 spot Linked Role
This step is only necessary if this is the first time you’re using EC2 Spot in this account. It is necessary to exist in your account in order to let you launch Spot instances.
aws iam create-service-linked-role --aws-service-name spot.amazonaws.comConfiguring the EKS Cluster !!
Next we will be deploying the cluster itself first , which has deployed a minimum set of On-Demand instances that we will use to deploy Kubernetes controllers on it.
After that we will use Karpenter to deploy Spot instances to showcase a few of the benefits of running a group-less auto scaler.
Terraform code for Setting up the entire stack can be found at :
https://github.com/NITHIN-JOHN-GEORGE/eks-karpenter-controllers-spot-terraform
Creating Terraform files For the Setup !!
Provider Configuration
terraform {
required_version = "~> 1.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.16.0"
}kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.18.0"
} kubectl = {
source = "gavinbunney/kubectl"
version = ">= 1.7.0"
} helm = {
source = "hashicorp/helm"
version = "2.8.0"
}
}
}
provider "aws" {
region = var.region
access_key = var.AWS_ACCESS_KEY
secret_key = var.AWS_SECRET_KEY}output "endpoint" {
value = data.aws_eks_cluster.cluster.endpoint
}# output "kubeconfig-certificate-authority-data" {
# value = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
# }provider "kubernetes" {
host = data.aws_eks_cluster.cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.cluster.token
}provider "kubectl" {
host = data.aws_eks_cluster.cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.cluster.token
load_config_file = false
}provider "helm" {
kubernetes {
host = data.aws_eks_cluster.cluster.endpoint
token = data.aws_eks_cluster_auth.cluster.token
cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
}
}
EKS Cluster Setup
locals {
env = ["prod", "qa", "dev"]
}
#------------Security group for eks-cluster--------------------------------------------------
resource "aws_security_group" "control_plane_sg" {
count = contains(local.env, var.env) ? 1 : 0
name = "k8s-control-plane-sg"
vpc_id = "${data.aws_vpc.vpc.id}"
tags = {
Name = "k8s-control-plane-sg"
vpc_id = "${data.aws_vpc.vpc.id}"
ManagedBy = "terraform"
Env = var.env
"karpenter.sh/discovery" = var.cluster_name
}
}
#----------------------Security group traffic rules for eks cluster------------------------------------------
## Ingress rule
resource "aws_security_group_rule" "control_plane_inbound" {
description = "Allow worker Kubelets and pods to receive communication from the cluster control plane"
security_group_id = element(aws_security_group.control_plane_sg.*.id,0)
type = "ingress"
from_port = 0
to_port = 65535
protocol = "tcp"
cidr_blocks = flatten(["${values(data.aws_subnet.public_subnets).*.cidr_block}", "${values(data.aws_subnet.private_subnets).*.cidr_block}"])
}
## Egress rule
resource "aws_security_group_rule" "control_plane_outbound" {
security_group_id = element(aws_security_group.control_plane_sg.*.id,0)
type = "egress"
from_port = 0
to_port = 65535
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
#------------Security group for eks-nodes--------------------------------------------------
resource "aws_security_group" "eks-nodes" {
count = contains(local.env, var.env) ? 1 : 0
name = "nodes_eks_sg"
description = "nodes_eks_sg"
vpc_id = "${data.aws_vpc.vpc.id}"
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Name = "nodes_eks_sg"
vpc_id = "${data.aws_vpc.vpc.id}"
ManagedBy = "terraform"
Env = var.env
"karpenter.sh/discovery" = var.cluster_name
}
}
#---------Security group traffic rules for node-security-group----------------
## Ingress rule
resource "aws_security_group_rule" "nodes_ssh" {
security_group_id = element(aws_security_group.eks-nodes.*.id,0)
type = "ingress"
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = flatten(["${values(data.aws_subnet.public_subnets).*.cidr_block}", "${values(data.aws_subnet.private_subnets).*.cidr_block}"])
}
#--------------------cloud-watch-log-group-- for eks cluster----------------------------------------------
resource "aws_cloudwatch_log_group" "eks_log_group" {
name = "/aws/eks/${var.cluster_name}/cluster"
retention_in_days = var.retention_day
tags = {
Name = "/aws/eks/${var.cluster_name}/cluster"
vpc_id = "${data.aws_vpc.vpc.id}"
ManagedBy = "terraform"
Env = var.env
}
}
#-----------------EKS-cluster-code--------------------------------------------
resource "aws_eks_cluster" "eks-cluster" {
count = contains(local.env, var.env) ? 1 : 0
name = "${var.cluster_name}"
enabled_cluster_log_types = ["api","audit","authenticator","controllerManager","scheduler"]
version = "${var.eks_version}"
role_arn = element(aws_iam_role.EKSClusterRole.*.arn,0)
vpc_config {
endpoint_private_access = true
endpoint_public_access = true
security_group_ids = ["${element(aws_security_group.control_plane_sg.*.id,0)}"]
subnet_ids = flatten(["${values(data.aws_subnet.private_subnets).*.id}"])
}
tags = {
Name = "${var.cluster_name}"
vpc_id = "${data.aws_vpc.vpc.id}"
ManagedBy = "terraform"
"karpenter.sh/discovery" = var.cluster_name
Env = var.env
}
depends_on = [
aws_cloudwatch_log_group.eks_log_group,
aws_iam_role_policy_attachment.eks-cluster-AmazonEKSClusterPolicy,
aws_iam_role_policy_attachment.eks-cluster-AmazonEKSServicePolicy,
]
}
#--------------eks-private-node-group----------------------------------------------
resource "aws_eks_node_group" "node-group-private" {
cluster_name = element(aws_eks_cluster.eks-cluster.*.name,0)
node_group_name = var.node_group_name
node_role_arn = element(aws_iam_role.NodeGroupRole.*.arn,0)
subnet_ids = flatten(["${values(data.aws_subnet.private_subnets).*.id}"])
capacity_type = "ON_DEMAND"
ami_type = var.ami_type
disk_size = var.disk_size
instance_types = var.instance_types
scaling_config {
desired_size = var.node_desired_size
max_size = var.node_max_size
min_size = var.node_min_size
}
lifecycle {
create_before_destroy = true
}
timeouts {}
remote_access {
ec2_ssh_key = var.ec2_ssh_key_name_eks_nodes
source_security_group_ids = [element(aws_security_group.eks-nodes.*.id,0)]
}
labels = {
"eks/cluster-name" = element(aws_eks_cluster.eks-cluster.*.name,0)
"eks/nodegroup-name" = format("nodegroup_%s", lower(element(aws_eks_cluster.eks-cluster.*.name,0)))
}
tags = merge({
Name = var.node_group_name
"eks/cluster-name" = element(aws_eks_cluster.eks-cluster.*.name,0)
"eks/nodegroup-name" = format("nodegroup_%s", lower(element(aws_eks_cluster.eks-cluster.*.name,0)))
"kubernetes.io/cluster/${var.cluster_name}" = "owned"
"karpenter.sh/discovery/${var.cluster_name}" = var.cluster_name
"karpenter.sh/discovery" = var.cluster_name
"eks/nodegroup-type" = "managed"
vpc_id = "${data.aws_vpc.vpc.id}"
ManagedBy = "terraform"
Env = var.env
})
# Ensure that IAM Role permissions are created before and deleted after EKS Node Group handling.
# Otherwise, EKS will not be able to properly delete EC2 Instances and Elastic Network Interfaces.
depends_on = [
aws_eks_cluster.eks-cluster,
aws_iam_role_policy_attachment.AmazonEKSWorkerNodePolicy,
aws_iam_role_policy_attachment.AmazonEKS_CNI_Policy,
aws_iam_role_policy_attachment.AmazonEC2ContainerRegistryReadOnly,
]
}
# ADD-ONS
resource "aws_eks_addon" "addons" {
for_each = { for addon in var.addons : addon.name => addon }
cluster_name = element(aws_eks_cluster.eks-cluster.*.name,0)
addon_name = each.value.name
resolve_conflicts = "OVERWRITE"
depends_on = [aws_eks_cluster.eks-cluster , aws_eks_node_group.node-group-private ]
}
IAM roles and Policies for EKS cluster and NodeGroup
# IAM role for EKS cluster
resource "aws_iam_role" "EKSClusterRole" {
count = contains(local.env, var.env) ? 1 : 0
name = "EKSClusterRole_v2"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "eks.amazonaws.com"
}
},
]
})
}
# Adding policies to the IAM role for EKS cluster
resource "aws_iam_role_policy_attachment" "eks-cluster-AmazonEKSClusterPolicy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
role = element(aws_iam_role.EKSClusterRole.*.name,0)
}
resource "aws_iam_role_policy_attachment" "eks-cluster-AmazonEKSVPCResourceController" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
role = element(aws_iam_role.EKSClusterRole.*.name,0)
}
resource "aws_iam_role_policy_attachment" "eks-cluster-AmazonEKSServicePolicy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
role = element(aws_iam_role.EKSClusterRole.*.name,0)
}
resource "aws_iam_role_policy_attachment" "eks_CloudWatchFullAccess" {
policy_arn = "arn:aws:iam::aws:policy/CloudWatchFullAccess"
role =element(aws_iam_role.EKSClusterRole.*.name,0)
}
## IAM role for Node group
resource "aws_iam_role" "NodeGroupRole" {
count = contains(local.env, var.env) ? 1 : 0
name = "EKSNodeGroupRole_v2"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
},
]
})
}
#---policy-attachements-for-Node group-role--------
resource "aws_iam_role_policy" "node-group-ClusterAutoscalerPolicy" {
name = "eks-cluster-auto-scaler"
role = element(aws_iam_role.NodeGroupRole.*.id,0)
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
]
Effect = "Allow"
Resource = "*"
},
]
})
}
resource "aws_iam_role_policy_attachment" "node_group_AWSLoadBalancerControllerPolicy" {
policy_arn = element(aws_iam_policy.load-balancer-policy.*.arn,0)
role = element(aws_iam_role.NodeGroupRole.*.name,0)
}
resource "aws_iam_role_policy_attachment" "AmazonEKSWorkerNodePolicy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
role = element(aws_iam_role.NodeGroupRole.*.name,0)
}
resource "aws_iam_role_policy_attachment" "AmazonEKS_CNI_Policy" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
role = element(aws_iam_role.NodeGroupRole.*.name,0)
}
resource "aws_iam_role_policy_attachment" "AmazonEC2ContainerRegistryReadOnly" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
role = element(aws_iam_role.NodeGroupRole.*.name,0)
}
resource "aws_iam_role_policy_attachment" "CloudWatchAgentServerPolicy" {
policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
role = element(aws_iam_role.NodeGroupRole.*.name,0)
}
## SSMManagedInstanceCore Policy for Nodes (Karpenter)
resource "aws_iam_role_policy_attachment" "eks_node_attach_AmazonSSMManagedInstanceCore" {
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
role = element(aws_iam_role.NodeGroupRole.*.name,0)
}
# Create IAM OIDC identity providers to establish trust between an OIDC-compatible IdP and your AWS account.
data "tls_certificate" "cert" {
url = aws_eks_cluster.eks-cluster[0].identity[0].oidc[0].issuer
}
resource "aws_iam_openid_connect_provider" "cluster" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.tls_certificate.cert.certificates[0].sha1_fingerprint]
url = aws_eks_cluster.eks-cluster[0].identity[0].oidc[0].issuer
}IAM role and policies for ALB ingress controller# IAM policy for ALB ingress controller
resource "aws_iam_policy" "load-balancer-policy" {
count = contains(local.env, var.env) ? 1 : 0
name = "AWSLoadBalancerControllerIAMPolicy_v2"
path = "/"
description = "AWS LoadBalancer Controller IAM Policy"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:CreateServiceLinkedRole"
],
"Resource": "*",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeAccountAttributes",
"ec2:DescribeAddresses",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInternetGateways",
"ec2:DescribeVpcs",
"ec2:DescribeVpcPeeringConnections",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeInstances",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeTags",
"ec2:GetCoipPoolUsage",
"ec2:DescribeCoipPools",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeListenerCertificates",
"elasticloadbalancing:DescribeSSLPolicies",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:DescribeTags"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"cognito-idp:DescribeUserPoolClient",
"acm:ListCertificates",
"acm:DescribeCertificate",
"iam:ListServerCertificates",
"iam:GetServerCertificate",
"waf-regional:GetWebACL",
"waf-regional:GetWebACLForResource",
"waf-regional:AssociateWebACL",
"waf-regional:DisassociateWebACL",
"wafv2:GetWebACL",
"wafv2:GetWebACLForResource",
"wafv2:AssociateWebACL",
"wafv2:DisassociateWebACL",
"shield:GetSubscriptionState",
"shield:DescribeProtection",
"shield:CreateProtection",
"shield:DeleteProtection"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateSecurityGroup"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateTags"
],
"Resource": "arn:aws:ec2:*:*:security-group/*",
"Condition": {
"StringEquals": {
"ec2:CreateAction": "CreateSecurityGroup"
},
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "false"
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateTags",
"ec2:DeleteTags"
],
"Resource": "arn:aws:ec2:*:*:security-group/*",
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress",
"ec2:DeleteSecurityGroup"
],
"Resource": "*",
"Condition": {
"Null": {
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
}
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateTargetGroup"
],
"Resource": "*",
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "false"
}
}
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:CreateRule",
"elasticloadbalancing:DeleteRule"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:AddTags",
"elasticloadbalancing:RemoveTags"
],
"Resource": [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
],
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
}
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:AddTags",
"elasticloadbalancing:RemoveTags"
],
"Resource": [
"arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
]
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:SetSubnets",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:DeleteTargetGroup"
],
"Resource": "*",
"Condition": {
"Null": {
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
}
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets"
],
"Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*"
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:SetWebAcl",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:AddListenerCertificates",
"elasticloadbalancing:RemoveListenerCertificates",
"elasticloadbalancing:ModifyRule"
],
"Resource": "*"
}
]
}
EOF
}
# IAM role and policy document for ALB ingress controller
data "aws_iam_policy_document" "eks_oidc_assume_role" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
effect = "Allow"
condition {
test = "StringEquals"
variable = "${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}:sub"
values = [
"system:serviceaccount:kube-system:aws-load-balancer-controller"
]
}
condition {
test = "StringEquals"
variable = "${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}:aud"
values = [
"sts.amazonaws.com"
]
}
principals {
identifiers = [
"arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}"
]
type = "Federated"
}
}
}
resource "aws_iam_role" "alb_ingress" {
count = contains(local.env, var.env) ? 1 : 0
name = "${var.cluster_name}-alb-ingress"
assume_role_policy = data.aws_iam_policy_document.eks_oidc_assume_role.json
}
resource "aws_iam_role_policy_attachment" "load-balancer-policy-role" {
policy_arn = element(aws_iam_policy.load-balancer-policy.*.arn,0)
role = element(aws_iam_role.alb_ingress.*.name,0)
}IAM roles and policies for EBS CSI driver
data "aws_iam_policy_document" "eks_oidc_assume_role_ebs" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
effect = "Allow"
condition {
test = "StringEquals"
variable = "${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}:sub"
values = [
"system:serviceaccount:kube-system:ebs-csi-controller-sa"
]
}
principals {
identifiers = [
"arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}"
]
type = "Federated"
}
}
}
resource "aws_iam_role" "ebs-csi" {
name = "ebs-csi-role"
assume_role_policy = data.aws_iam_policy_document.eks_oidc_assume_role_ebs.json
}
# IAM policy for EBS CSI controller
resource "aws_iam_policy" "ebs_permissions" {
name = "ebs-permissions"
path = "/"
description = "AWS EBS Controller IAM Policy"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:CreateSnapshot",
"ec2:AttachVolume",
"ec2:DetachVolume",
"ec2:ModifyVolume",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeInstances",
"ec2:DescribeSnapshots",
"ec2:DescribeTags",
"ec2:DescribeVolumes",
"ec2:DescribeVolumesModifications"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateTags"
],
"Resource": [
"arn:aws:ec2:*:*:volume/*",
"arn:aws:ec2:*:*:snapshot/*"
],
"Condition": {
"StringEquals": {
"ec2:CreateAction": [
"CreateVolume",
"CreateSnapshot"
]
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:DeleteTags"
],
"Resource": [
"arn:aws:ec2:*:*:volume/*",
"arn:aws:ec2:*:*:snapshot/*"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateVolume"
],
"Resource": "*",
"Condition": {
"StringLike": {
"aws:RequestTag/ebs.csi.aws.com/cluster": "true"
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateVolume"
],
"Resource": "*",
"Condition": {
"StringLike": {
"aws:RequestTag/CSIVolumeName": "*"
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:DeleteVolume"
],
"Resource": "*",
"Condition": {
"StringLike": {
"ec2:ResourceTag/CSIVolumeName": "*"
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:DeleteVolume"
],
"Resource": "*",
"Condition": {
"StringLike": {
"ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:DeleteSnapshot"
],
"Resource": "*",
"Condition": {
"StringLike": {
"ec2:ResourceTag/CSIVolumeSnapshotName": "*"
}
}
},
{
"Effect": "Allow",
"Action": [
"ec2:DeleteSnapshot"
],
"Resource": "*",
"Condition": {
"StringLike": {
"ec2:ResourceTag/ebs.csi.aws.com/cluster": "true"
}
}
}
]
}
EOF
}
resource "aws_iam_role_policy_attachment" "ebs-csi-policy-role" {
policy_arn = aws_iam_policy.ebs_permissions.arn
role = aws_iam_role.ebs-csi.name
}Template Files ( Custom values.yml for helm chart ) FOR ALB-INGRESS AND EBS-CSI DRIVER
data "template_file" "alb-ingress-values" {
template = <<EOF
replicaCount: 1vpcId: "${data.aws_vpc.vpc.id}"
clusterName: "${var.cluster_name}"
ingressClass: alb
createIngressClassResource: true
region: "${var.region}"
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 512Mi
cpu: 1000m
EOF
}
data "template_file" "ebs-csi-driver-values" {
template = <<EOF
controller:
region: "${var.region}"
replicaCount: 1
k8sTagClusterId: "${var.cluster_name}"
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
resources:
requests:
cpu: 10m
memory: 40Mi
limits:
cpu: 100m
memory: 256Mi
serviceAccount:
create: true
name: ebs-csi-controller-sa
annotations:
eks.amazonaws.com/role-arn: "${aws_iam_role.ebs-csi.arn}"
storageClasses:
- name: ebs-sc
annotations:
storageclass.kubernetes.io/is-default-class: "true"
EOF
}
Kuberenets Resources for ALB ingress controller
resource "kubernetes_service_account" "aws-load-balancer-controller-service-account" {
metadata {
name = "aws-load-balancer-controller"
namespace = "kube-system"
annotations = {
"eks.amazonaws.com/role-arn" = data.aws_iam_role.alb_ingress.arn
}
labels = {
"app.kubernetes.io/name" = "aws-load-balancer-controller"
"app.kubernetes.io/component" = "controller"
"app.kubernetes.io/managed-by" = "terraform"
}
}
automount_service_account_token = true
depends_on = [aws_eks_cluster.eks-cluster , aws_eks_node_group.node-group-private ]
}
resource "kubernetes_secret" "aws-load-balancer-controller" {
metadata {
name = "aws-load-balancer-controller"
namespace = "kube-system"
annotations = {
"kubernetes.io/service-account.name" = "aws-load-balancer-controller"
"kubernetes.io/service-account.namespace" = "kube-system"
}
}
type = "kubernetes.io/service-account-token"
depends_on = [kubernetes_service_account.aws-load-balancer-controller-service-account , aws_eks_cluster.eks-cluster , aws_eks_node_group.node-group-private]
}
resource "kubernetes_cluster_role" "aws-load-balancer-controller-cluster-role" {
depends_on = [aws_eks_cluster.eks-cluster , aws_eks_node_group.node-group-private ]
metadata {
name = "aws-load-balancer-controller"
labels = {
"app.kubernetes.io/name" = "aws-load-balancer-controller"
"app.kubernetes.io/managed-by" = "terraform"
}
}
rule {
api_groups = [
"",
"extensions",
]
resources = [
"configmaps",
"endpoints",
"events",
"ingresses",
"ingresses/status",
"services",
]
verbs = [
"create",
"get",
"list",
"update",
"watch",
"patch",
]
}
rule {
api_groups = [
"",
"extensions",
]
resources = [
"nodes",
"pods",
"secrets",
"services",
"namespaces",
]
verbs = [
"get",
"list",
"watch",
]
}
}
resource "kubernetes_cluster_role_binding" "aws-load-balancer-controller-cluster-role-binding" {
depends_on = [aws_eks_cluster.eks-cluster , aws_eks_node_group.node-group-private ]
metadata {
name = "aws-load-balancer-controller"
labels = {
"app.kubernetes.io/name" = "aws-load-balancer-controller"
"app.kubernetes.io/managed-by" = "terraform"
}
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = kubernetes_cluster_role.aws-load-balancer-controller-cluster-role.metadata[0].name
}
subject {
api_group = ""
kind = "ServiceAccount"
name = kubernetes_service_account.aws-load-balancer-controller-service-account.metadata[0].name
namespace = kubernetes_service_account.aws-load-balancer-controller-service-account.metadata[0].namespace
}
}Helm Release for ALB ingress controller and EBS CSI driver and Metrics Server
resource "helm_release" "alb-ingress-controller" {
depends_on = [
aws_eks_cluster.eks-cluster ,
aws_eks_node_group.node-group-private ,
kubernetes_cluster_role_binding.aws-load-balancer-controller-cluster-role-binding,
kubernetes_service_account.aws-load-balancer-controller-service-account ,
kubernetes_secret.aws-load-balancer-controller ,
helm_release.karpenter ,
kubectl_manifest.karpenter-provisioner ]
name = "alb-ingress-controller"
repository = "<https://aws.github.io/eks-charts>"
version = "1.4.7"
chart = "aws-load-balancer-controller"
namespace = "kube-system"
values = [data.template_file.alb-ingress-values.rendered]
set {
name = "serviceAccount.create"
value = "false"
}
set {
name = "serviceAccount.name"
value = "${kubernetes_service_account.aws-load-balancer-controller-service-account.metadata[0].name}"
}
}
resource "helm_release" "aws-ebs-csi-driver" {
depends_on = [ aws_eks_cluster.eks-cluster , aws_eks_node_group.node-group-private , helm_release.karpenter , kubectl_manifest.karpenter-provisioner , aws_iam_role.ebs-csi ]
name = "aws-ebs-csi-driver"
repository = "<https://kubernetes-sigs.github.io/aws-ebs-csi-driver>"
chart = "aws-ebs-csi-driver"
namespace = "kube-system"
create_namespace = true
values = [data.template_file.ebs-csi-driver-values.rendered]
}
resource "helm_release" "metrics-server" {
depends_on = [helm_release.karpenter , kubectl_manifest.karpenter-provisioner ]
name = "metrics-server"
chart = "metrics-server"
repository = "<https://kubernetes-sigs.github.io/metrics-server/>"
version = "3.8.2"
namespace = "kube-system"
description = "Metric server helm Chart deployment configuration"
}IAM role and policies for Karpenter controller
data "aws_iam_policy_document" "karpenter_controller_assume_role_policy" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
effect = "Allow"
condition {
test = "StringEquals"
variable = "${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}:sub"
values = ["system:serviceaccount:karpenter:karpenter"]
}
condition {
test = "StringEquals"
variable = "${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}:aud"
values = ["sts.amazonaws.com"]
}
principals {
identifiers = [
"arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/${replace(data.aws_eks_cluster.cluster.identity[0].oidc[0].issuer, "https://", "")}"
]
type = "Federated"
}
}
}
data "aws_iam_policy_document" "karpenter" {
statement {
resources = ["*"]
actions = ["ec2:DescribeImages", "ec2:RunInstances", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "ec2:DescribeLaunchTemplates", "ec2:DescribeInstances", "ec2:DescribeInstanceTypes", "ec2:DescribeInstanceTypeOfferings", "ec2:DescribeAvailabilityZones", "ec2:DeleteLaunchTemplate", "ec2:CreateTags", "ec2:CreateLaunchTemplate", "ec2:CreateFleet", "ec2:DescribeSpotPriceHistory", "pricing:GetProducts", "ssm:GetParameter"]
effect = "Allow"
}
statement {
resources = ["*"]
actions = ["ec2:TerminateInstances", "ec2:DeleteLaunchTemplate" , "ec2:RequestSpotInstances" , "ec2:DescribeInstanceStatus" , "iam:CreateServiceLinkedRole" , "iam:ListRoles" , "iam:ListInstanceProfiles"]
effect = "Allow"
# Make sure Karpenter can only delete nodes that it has provisioned
condition {
test = "StringEquals"
values = [var.eks_cluster_name]
variable = "ec2:ResourceTag/karpenter.sh/discovery"
}
}
statement {
resources = [data.aws_eks_cluster.cluster.arn]
actions = ["eks:DescribeCluster"]
effect = "Allow"
}
statement {
resources = [element(aws_iam_role.NodeGroupRole.*.arn,0)]
actions = ["iam:PassRole"]
effect = "Allow"
}
# Optional: Interrupt Termination Queue permissions, provided by AWS SQS
statement {
resources = [aws_sqs_queue.karpenter.arn]
actions = ["sqs:DeleteMessage", "sqs:GetQueueUrl", "sqs:GetQueueAttributes", "sqs:ReceiveMessage"]
effect = "Allow"
}
}
resource "aws_iam_role" "karpenter_controller" {
description = "IAM Role for Karpenter Controller (pod) to assume"
assume_role_policy = data.aws_iam_policy_document.karpenter_controller_assume_role_policy.json
name = "karpenter-controller"
inline_policy {
policy = data.aws_iam_policy_document.karpenter.json
name = "karpenter"
}
depends_on = [data.aws_iam_policy_document.karpenter_controller_assume_role_policy , data.aws_iam_policy_document.karpenter]
}Instance Profile for Karpenter
## Karpenter Instance Profile
resource "aws_iam_instance_profile" "karpenter" {
name = "karpenter-instance-profile"
role = element(aws_iam_role.NodeGroupRole.*.name,0)
depends_on = [ aws_iam_role.NodeGroupRole ]
}Handling Interruption For Spot Instance via SQS queue and Event Bridge rules
# SQS Queue
resource "aws_sqs_queue" "karpenter" {
message_retention_seconds = 300
name = "${var.cluster_name}-karpenter-sqs-queue"
}
# Node termination queue policy
resource "aws_sqs_queue_policy" "karpenter" {
policy = data.aws_iam_policy_document.node_termination_queue.json
queue_url = aws_sqs_queue.karpenter.url
}
data "aws_iam_policy_document" "node_termination_queue" {
statement {
resources = [aws_sqs_queue.karpenter.arn]
sid = "SQSWrite"
actions = ["sqs:SendMessage"]
principals {
type = "Service"
identifiers = ["events.amazonaws.com", "sqs.amazonaws.com"]
}
}
}
resource "aws_cloudwatch_event_rule" "scheduled_change_rule" {
name = "ScheduledChangeRule"
description = "AWS Health Event"
event_pattern = jsonencode({
source = ["aws.health"]
detail_type = ["AWS Health Event"]
})
}
resource "aws_cloudwatch_event_rule" "spot_interruption_rule" {
name = "SpotInterruptionRule"
description = "EC2 Spot Instance Interruption Warning"
event_pattern = jsonencode({
source = ["aws.ec2"]
detail_type = ["EC2 Spot Instance Interruption Warning"]
})
}
resource "aws_cloudwatch_event_rule" "rebalance_rule" {
name = "RebalanceRule"
description = "EC2 Instance Rebalance Recommendation"
event_pattern = jsonencode({
source = ["aws.ec2"]
detail_type = ["EC2 Instance Rebalance Recommendation"]
})
}
resource "aws_cloudwatch_event_rule" "instance_state_change_rule" {
name = "InstanceStateChangeRule"
description = "EC2 Instance State-change Notification"
event_pattern = jsonencode({
source = ["aws.ec2"]
detail_type = ["EC2 Instance State-change Notification"]
})
}
resource "aws_cloudwatch_event_target" "scheduled_change_rule" {
rule = aws_cloudwatch_event_rule.scheduled_change_rule.name
arn = aws_sqs_queue.karpenter.arn
}
resource "aws_cloudwatch_event_target" "spot_interruption_rule" {
rule = aws_cloudwatch_event_rule.spot_interruption_rule.name
arn = aws_sqs_queue.karpenter.arn
}
resource "aws_cloudwatch_event_target" "rebalance_rule" {
rule = aws_cloudwatch_event_rule.rebalance_rule.name
arn = aws_sqs_queue.karpenter.arn
}
resource "aws_cloudwatch_event_target" "instance_state_change_rule" {
rule = aws_cloudwatch_event_rule.instance_state_change_rule.name
arn = aws_sqs_queue.karpenter.arn
}Template File ( values.yml ) for Karpenter
data "template_file" "karpenter" {
template = <<EOF
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "${aws_iam_role.karpenter_controller.arn}"
settings:
aws:
clusterName: "${data.aws_eks_cluster.cluster.id}"
clusterEndpoint: "${data.aws_eks_cluster.cluster.endpoint}"
defaultInstanceProfile: "${aws_iam_instance_profile.karpenter.name}"
interruptionQueueName: "${var.cluster_name}-karpenter-sqs-queue"
EOF
}Helm release for Karpenter
resource "helm_release" "karpenter" {
namespace = "karpenter"
create_namespace = true
name = "karpenter"
repository = "oci://public.ecr.aws/karpenter"
repository_username = data.aws_ecrpublic_authorization_token.token.user_name
repository_password = data.aws_ecrpublic_authorization_token.token.password
chart = "karpenter"
version = "v0.27.0"
values = [data.template_file.karpenter.rendered]
# set {
# name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
# value = "${aws_iam_role.karpenter_controller.arn}"
# }
# set {
# name = "settings.aws.clusterName"
# value = data.aws_eks_cluster.cluster.id
# }
# set {
# name = "settings.aws.clusterEndpoint"
# value = data.aws_eks_cluster.cluster.endpoint
# }
# set {
# name = "settings.aws.defaultInstanceProfile"
# value = aws_iam_instance_profile.karpenter.name
# }
# set {
# name = "settings.aws.interruptionQueueName"
# value = "${var.cluster_name}-karpenter-sqs-queue"
# }
depends_on = [ aws_eks_cluster.eks-cluster , aws_eks_node_group.node-group-private , aws_iam_role.karpenter_controller , aws_iam_instance_profile.karpenter , aws_sqs_queue.karpenter ]
}Not finished yet !! We need
Provisionersinstalled which decides what type of instance to bring up based on the workload.
Provisioners are CRDs that Karpenter uses to provision new nodes. New nodes are brought up based on the pods that are waiting to be scheduled and their scheduling constraints
Configuring Default Karpenter Provisioner
resource "kubectl_manifest" "karpenter-provisioner" {
depends_on = [ helm_release.karpenter , aws_eks_cluster.eks-cluster , aws_eks_node_group.node-group-private]
yaml_body = <<YAML
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: karpenter-default
namespace: karpenter
spec:
provider:
securityGroupSelector:
karpenter.sh/discovery: "${var.cluster_name}"
subnetSelector:
karpenter.sh/discovery: "${var.cluster_name}"
tags:
karpenter.sh/discovery: "${var.cluster_name}"
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
- key: "node.kubernetes.io/instance-type"
operator: In
values: ["t3a.micro" , "t3a.medium" , "t3a.large" ]
- key: "topology.kubernetes.io/zone"
operator: In
values: ["us-east-1a", "us-east-1b" , "us-east-1c"]
- key: created-by
operator: In
values: ["karpenter"]
ttlSecondsAfterEmpty: 30
YAML
}Note: securityGroupSelector and subnetSelector — These are used by Karpenter to identify which subnets and security groups should be used for the new nodes. When you bring up your EKS cluster you should add the respective tags to your private_subnets and to your node_security_group. Not setting these correctly can cause Karpenter to not bring up nodes or the nodes that are brought up can have difficulties trying to join the cluster.
The constraints ttlSecondsUntilExpired defines the node expiry so a newer node will be provisioned, and ttlSecondsAfterEmpty defines when to delete a node since the last workload stops running. (Note: DaemonSets are not taken into account.)
Deploying via Terraform Cloud !!!
I am deploying the EKS Cluster to the AWS via Terraform Cloud . I have workspaces and made connections to the repository and updated variables.
Workspace Run Overview:
Workspace Variables Overview:
Workspace Planned:
Workspace Applied:
Testing it !! Its Time for Action
Updating the Kubeconfig File
aws eks update-kubeconfig --region us-east-1 --name karpenter-demoListing all Pods in the Cluster
kubectl get pods -A
Listing all resources in Karpenter Namespace
kubectl create deployment nginx --image=nginx
kubectl set resources deployment nginx --requests=cpu=100m,memory=256Mi
kubectl scale deployment nginx --replicas 12
We can see a new node got provisioned . To verify that it created a spot instance lets go to the console and see:
Now lets scale down to see what happens.
kubectl scale deployment nginx --replicas 1Immediately after that :
Lets check console whats happening.
This confirms our setup was successfully completed !!
Thank you for reading my article , have a nice day and keep learning!
