Karpenter: A Deployment Walkthrough
If you haven’t had a chance to get to know Karpenter, now it’s the best chance to take a look at my previous article: Karpenter — AWS Flying Magic Carpet
In this article, I’m presuming that you have deployed the EKS cluster with the help of my previous EKS Cluster: A Deployment Walkthrough article, but if not — everything will be pretty easy to integrate if you are using an official AWS EKS module.
In case you find yourself uncertain, all Terraform configuration files related to this article are available on my GitHub repository:
All the resources should be applied only after we’ve defined all the resources in the directory structure I’ve outlined below. Please be sure that you have already created EKS cluster before getting further.
Note: Directory structure is from eks-karpenter repository. The only file you will need to create is a karpenter.tf . A very small amount of changes will be done to eks.tf, providers.tf, and version.tf configuration files.
.
└── terraform/
├── eks.tf (will be modified)
├── iam_roles.tf
├── kms.tf
├── karpenter.tf (new file)
├── locals.tf
├── main.tf
├── kms.tf
├── providers.tf (will be modified)
├── variables.tf
└── version.tf (will be modified)Before getting into setting up needed resources — we need to add some additional providers that will be used in our setup. This is how the version.tf should look like after adding helm and kubectl providers:
terraform {
required_version = ">= 1.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 4.47"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.10"
}
###########################################
## Helm and Kubectl providers were added ##
###########################################
helm = {
source = "hashicorp/helm"
version = ">= 2.7"
}
kubectl = {
source = "gavinbunney/kubectl"
version = ">= 1.14"
}
###########################################
null = {
source = "hashicorp/null"
version = ">= 3.0"
}
}
}In the providers.tf file we are adding authentication values for a helm provider. These are the same data variables that we used to configure Kubernetes provider in the same providers.tf file:
provider "helm" {
kubernetes {
host = data.aws_eks_cluster.cluster.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.this.token
}
}After configuring an additional provider, we will proceed to add a first configuration block in our karpenter.tf file. This configuration block includes official AWS Karpenter module, which will deploy all the necessary IAM resources required for Karpenter to operate. And yes, you won’t need to manually create any IAM resources. That brought me a lot of joy.
Note: This is a description of the exact resources that the Karpenter module will create for you:
#################################################
### Node termination event related resources ###
#################################################
aws_cloudwatch_event_rule.this - Creates an Event Rule that checks for events from SQS.
aws_cloudwatch_event_target.this - Creates a target that links to SQS queues.
aws_sqs_queue.this - Karpenter is using the SQS queue to watch AWS Health and Spot instance interruption events.
aws_sqs_queue_policy.this - Policy to allow sending SQS messages.
#####################################################
### IAM Role for Service Account (IRSA) Resources ###
### This is used by the Karpenter controller ###
#####################################################
aws_iam_policy.irsa - Policy to allow to do EC2 specific actions through AWS API for IRSA.
aws_iam_role.irsa - IRSA role creation.
aws_iam_role_policy_attachment.irsa - Attachment of IRSA policy to IRSA Role.
aws_iam_role_policy_attachment.irsa_additional - Attachment of additional user-specified policy to IRSA Role.
########################################################
### Node IAM Role ###
### This is used by the nodes launched by Karpenter ###
########################################################
aws_iam_instance_profile.this - Instance profile with attached roles.
aws_iam_role.this - Role that will be used by Nodes.
aws_iam_role_policy_attachment.additional - User specified additional policy
aws_iam_role_policy_attachment.this - User specified an additional policy.And here is the actual module configuration block that you should add to your karpenter.tf file:
# This module creates needed IAM resources which we will use when deploying Karpenter resources with Helm
module "karpenter" {
version = "v19.16.0"
source = "terraform-aws-modules/eks/aws//modules/karpenter"
cluster_name = module.eks_cluster.cluster_name
irsa_oidc_provider_arn = module.eks_cluster.oidc_provider_arn
irsa_namespace_service_accounts = ["karpenter:karpenter"]
create_iam_role = false
iam_role_arn = module.eks_cluster.eks_managed_node_groups["initial"].iam_role_arn
policies = {
AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
tags = {
Environment = "dev"
Terraform = "true"
}After dealing with IAM stuff — we need to define a helm chart, that will deploy Karpenter related resources within our Kubernetes cluster. Add this in karpenter.tf file:
# Provider and data resources will help us to mitigate AWS Public ECR related bug:
# https://github.com/aws/karpenter/issues/3015
provider "aws" {
region = "us-east-1"
alias = "virginia"
}
data "aws_ecrpublic_authorization_token" "token" {
provider = aws.virginia
}
resource "helm_release" "karpenter" {
namespace = "karpenter"
create_namespace = true
# We will be using AWS Public ECR to download needed chart as mentioned Above.
name = "karpenter"
repository = "oci://public.ecr.aws/karpenter"
repository_username = data.aws_ecrpublic_authorization_token.token.user_name
repository_password = data.aws_ecrpublic_authorization_token.token.password
chart = "karpenter"
version = "v0.21.1"
# We are setting the cluster name of our already created EKS cluster
set {
name = "settings.aws.clusterName"
value = module.eks_cluster.cluster_name
}
# Setting EKS Endpoint URL which looks like https://<Gibberish>.yl4.eu-central-1.eks.amazonaws.com
set {
name = "settings.aws.clusterEndpoint"
value = module.eks_cluster.cluster_endpoint
}
# We will be using IRSA role ARN which we created previously with Karpenter module
set {
name = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
value = module.karpenter.irsa_arn
}
# We will be using instance profile name which we created previously with Karpenter module
set {
name = "settings.aws.defaultInstanceProfile"
value = module.karpenter.instance_profile_name
}
# We will be using SQS queue name which we created previously with Karpenter module
set {
name = "settings.aws.interruptionQueueName"
value = module.karpenter.queue_name
}
}Next, we will create a tag for a security group that is utilized during the deployment of worker nodes within the EKS cluster. Add an additional node_security_group_tags block in the eks.tf file. This is how it should look like:
<...truncated...>
node_security_group_additional_rules = {
node_to_node_ig = {
description = "Node to node ingress traffic"
from_port = 1
to_port = 65535
protocol = "all"
type = "ingress"
self = true
}
}
# With this tag - karpenter will detect which security group to use for autoscaled nodes
node_security_group_tags = {
"karpenter.sh/discovery" = "opsbridge-cluster"
}
}After tagging a security group — we need to tag subnets that Karpenter will use when creating additional nodes. Go into your AWS Console > VPC > Subnets
In your selected subnet create a tag that will contain the key and a value similar to this example:
key: karpenter.sh/discovery
value: opsbridge-clusterOnce the necessary tags are added, the Karpenter Provisioner should be able to identify the security groups and subnets without encountering any issues.
And now we will be deploying Provisioner and NodeTemplate resources using the Kubernetes manifests.
Note: Apply the Kubernetes manifest resources only after you have successfully created an EKS cluster. Otherwise, you might encounter a “Failed to construct REST client” error.
When it comes to the Provisioner, feel free to modify the requirements in any way you prefer. You can refer to the examples provided in: https://karpenter.sh/preview/concepts/provisioners
Add a Provisioner manifest in karpenter.tf:
# Creating a provisioner which will create additional nodes for unscheduled pods
resource "kubernetes_manifest" "karpenter_provisioner" {
# Terraform by default doesn't tolerate values changing between configuration and apply results.
# Users are required to declare these tolerable exceptions explicitly.
# With a kubernetes_manifest resource, you can achieve this by using the computed_fields meta-attribute.
computed_fields = ["spec.requirements", "spec.limits"]
manifest = yamldecode(<<-EOF
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: karpenter.sh/capacity-type
operator: In
values: ["spot"]
- key: "karpenter.k8s.aws/instance-category"
operator: In
values: ["t"]
- key: "karpenter.k8s.aws/instance-cpu"
operator: In
values: ["2"]
- key: "karpenter.k8s.aws/instance-generation"
operator: In
values: ["3"]
limits:
resources:
cpu: 1000
providerRef:
name: default
ttlSecondsAfterEmpty: 30
EOF
)
depends_on = [
helm_release.karpenter
]
}Add a NodeTemplate manifest in karpenter.tf:
# Creating a Node template, which will be used for Node configuration in AWS side
resource "kubernetes_manifest" "karpenter_node_template" {
# Terraform by default doesn't tolerate values changing between configuration and apply results.
# Users are required to declare these tolerable exceptions explicitly.
# With a kubernetes_manifest resource, you can achieve this by using the computed_fields meta-attribute.
computed_fields = ["spec.requirements", "spec.limits"]
manifest = yamldecode(<<-EOF
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
name: default
spec:
subnetSelector:
karpenter.sh/discovery: ${module.eks_cluster.cluster_name}
securityGroupSelector:
karpenter.sh/discovery: ${module.eks_cluster.cluster_name}
tags:
karpenter.sh/discovery: ${module.eks_cluster.cluster_name}
EOF
)
depends_on = [
helm_release.karpenter
]
}After applying everything with terraform apply — you will see the resources created:
$ kubectl get all -n karpenter
NAME READY STATUS RESTARTS AGE
pod/karpenter-556f8d8f6b-94fkc 0/1 Pending 0 28s
pod/karpenter-556f8d8f6b-t4jvv 1/1 Running 0 6m31s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/karpenter ClusterIP 172.20.146.28 <none> 8080/TCP,443/TCP 6m31s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/karpenter 1/2 2 1 6m32s
NAME DESIRED CURRENT READY AGE
replicaset.apps/karpenter-556f8d8f6b 2 2 1 6m32s
$ kubectl get configmap -n karpenter
NAME DATA AGE
config-logging 2 9m28s
karpenter-global-settings 11 9m28s
kube-root-ca.crt 1 9m28s
$ kubectl describe configmap karpenter-global-settings -n karpenter
Name: karpenter-global-settings
Namespace: karpenter
Labels: app.kubernetes.io/instance=karpenter
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=karpenter
app.kubernetes.io/version=0.21.1
helm.sh/chart=karpenter-v0.21.1
Annotations: meta.helm.sh/release-name: karpenter
meta.helm.sh/release-namespace: karpenter
Data
====
aws.enablePodENI:
----
false
aws.isolatedVPC:
----
false
batchIdleDuration:
----
1s
batchMaxDuration:
----
10s
aws.clusterEndpoint:
----
https://<giberrish>.yl4.eu-central-1.eks.amazonaws.com
aws.clusterName:
----
opsbridge-cluster
aws.defaultInstanceProfile:
----
Karpenter-opsbridge-cluster-20230821
aws.enableENILimitedPodDensity:
----
true
aws.interruptionQueueName:
----
Karpenter-opsbridge-cluster
aws.nodeNameConvention:
----
ip-name
aws.vmMemoryOverheadPercent:
----
0.075
BinaryData
====
Events: <none>
$ kubectl describe provisioner
Name: default
Namespace:
Labels: <none>
Annotations: <none>
API Version: karpenter.sh/v1alpha5
Kind: Provisioner
Metadata:
Creation Timestamp: 2023-08-26T11:05:34Z
Generation: 1
Resource Version: 2734
UID: a7fa1ed7-27bd-1
Spec:
Limits:
Resources:
Cpu: 1k
Provider Ref:
Name: default
Requirements:
Key: karpenter.sh/capacity-type
Operator: In
Values:
spot
Key: karpenter.k8s.aws/instance-category
Operator: In
Values:
t
Key: karpenter.k8s.aws/instance-cpu
Operator: In
Values:
2
Key: karpenter.k8s.aws/instance-generation
Operator: In
Values:
3
Key: kubernetes.io/os
Operator: In
Values:
linux
Key: kubernetes.io/arch
Operator: In
Values:
amd64
Ttl Seconds After Empty: 30
Events: <none>
$ kubectl describe AWSNodeTemplate
Name: default
Namespace:
Labels: <none>
Annotations: <none>
API Version: karpenter.k8s.aws/v1alpha1
Kind: AWSNodeTemplate
Metadata:
Creation Timestamp: 2023-08-26T11:05:34Z
Generation: 1
Resource Version: 2735
UID: b0c99de0-73ab-1
Spec:
Security Group Selector:
karpenter.sh/discovery: opsbridge-cluster
Subnet Selector:
karpenter.sh/discovery: opsbridge-cluster
Tags:
karpenter.sh/discovery: opsbridge-cluster
Events: <none>Now we will be testing out if Karpenter autoscaling is working as it should:
Add an additional manifest in karpenter.tf and deploy it with terraform apply:
# Test deployment by using the [pause image](https://www.ianlewis.org/en/almighty-pause-container)
# If you want to inflate - use kubectl scale --replicas=20 deployment/inflate
resource "kubernetes_manifest" "karpenter_example_deployment" {
manifest = yamldecode(<<-EOF
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: default
name: inflate
spec:
replicas: 1
selector:
matchLabels:
app: inflate
template:
metadata:
labels:
app: inflate
spec:
terminationGracePeriodSeconds: 0
containers:
- name: inflate
image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
resources:
requests:
cpu: 500m
EOF
)
depends_on = [
helm_release.karpenter
]
}Now you should see that we have successfully created a deployment:
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
inflate 1/1 1 1 17sAnd if you would scale the deployment with:
kubectl scale --replicas=12 deployment/inflateCertain pods may become stuck in the Pending state, leading the Provisioner to initiate the creation of new nodes.
Before scaling the deployment — you will see that you have only one node:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-31-91-38.eu-central-1.compute.internal Ready <none> 25m v1.27.3-eks-a5565adOnce you’ve scaled the deployment, you should observe that some of the pods have been successfully deployed, while others remain in a pending status:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
inflate-bbd699657-45gh9 0/1 Pending 0 10s
inflate-bbd699657-4w2lv 0/1 Pending 0 10s
inflate-bbd699657-6qc7l 0/1 Pending 0 10s
inflate-bbd699657-6rnq6 0/1 Pending 0 10s
inflate-bbd699657-9nvsz 1/1 Running 0 2m44s
inflate-bbd699657-ltcll 0/1 Pending 0 10s
inflate-bbd699657-nnl9g 0/1 Pending 0 10s
inflate-bbd699657-ppz7j 0/1 Pending 0 10s
inflate-bbd699657-r7kw4 0/1 Pending 0 10s
inflate-bbd699657-tcvvw 0/1 Pending 0 10s
inflate-bbd699657-tghsg 0/1 Pending 0 10s
inflate-bbd699657-wgs4r 0/1 Pending 0 10sUpon inspecting the Karpenter Pod logs, you will notice that it has initiated the provisioning of new nodes:
$ kubectl logs -n karpenter karpenter-556f8d8f6b-55jg9
2023-08-26T11:31:10.038Z DEBUG controller.provisioner discovered subnets {"commit": "06cb81f-dirty", "subnets": ["<subnet-1> (eu-central-1a)"]}
2023-08-26T11:31:10.166Z DEBUG controller.provisioner discovered EC2 instance types zonal offerings for subnets {"commit": "06cb81f-dirty", "subnet-selector": "{\"karpenter.sh/discovery\":\"opsbridge-cluster\"}"}
2023-08-26T11:31:10.435Z INFO controller.provisioner found provisionable pod(s) {"commit": "06cb81f-dirty", "pods": 11}
2023-08-26T11:31:10.435Z INFO controller.provisioner computed new node(s) to fit pod(s) {"commit": "06cb81f-dirty", "newNodes": 4, "pods": 11}
2023-08-26T11:31:10.436Z INFO controller.provisioner launching node with 2 pods requesting {"cpu":"1125m","pods":"4"} from types t3.small, t3a.medium, t3.micro, t3.large, t3a.small and 3 other(s) {"commit": "06cb81f-dirty", "provisioner": "default"}
2023-08-26T11:31:10.445Z INFO controller.provisioner launching node with 3 pods requesting {"cpu":"1625m","pods":"5"} from types t3.small, t3a.medium, t3.large, t3a.small, t3a.large and 1 other(s) {"commit": "06cb81f-dirty", "provisioner": "default"}
2023-08-26T11:31:10.454Z INFO controller.provisioner launching node with 3 pods requesting {"cpu":"1625m","pods":"5"} from types t3.small, t3a.medium, t3.large, t3a.small, t3a.large and 1 other(s) {"commit": "06cb81f-dirty", "provisioner": "default"}
2023-08-26T11:31:10.455Z INFO controller.provisioner launching node with 3 pods requesting {"cpu":"1625m","pods":"5"} from types t3.small, t3a.medium, t3.large, t3a.small, t3a.large and 1 other(s) {"commit": "06cb81f-dirty", "provisioner": "default"}
2023-08-26T11:31:10.616Z DEBUG controller.provisioner.cloudprovider discovered security groups {"commit": "06cb81f-dirty", "provisioner": "default", "security-groups": ["sg-1"]}
2023-08-26T11:31:10.619Z DEBUG controller.provisioner.cloudprovider discovered kubernetes version {"commit": "06cb81f-dirty", "provisioner": "default", "kubernetes-version": "1.27"}
2023-08-26T11:31:10.661Z DEBUG controller.provisioner.cloudprovider discovered new ami {"commit": "06cb81f-dirty", "provisioner": "default", "ami": "ami-0561", "query": "/aws/service/eks/optimized-ami/1.27/amazon-linux-2/recommended/image_id"}
2023-08-26T11:31:10.783Z DEBUG controller.provisioner.cloudprovider created launch template {"commit": "06cb81f-dirty", "provisioner": "default", "launch-template-name": "Karpenter-opsbridge-cluster-5439092676711980080", "launch-template-id": "lt-007945e3a31c02826"}
2023-08-26T11:31:12.855Z INFO controller.provisioner.cloudprovider launched new instance {"commit": "06cb81f-dirty", "provisioner": "default", "launched-instance": "i-03498efaa087b3baf", "hostname": "ip-10-31-75-171.eu-central-1.compute.internal", "type": "t3.micro", "zone": "eu-central-1a", "capacity-type": "spot"}
2023-08-26T11:31:13.037Z ERROR controller Reconciler error {"commit": "06cb81f-dirty", "controller": "inflightchecks", "controllerGroup": "", "controllerKind": "Node", "Node": {"name":"ip-10-31-75-171.eu-central-1.compute.internal"}, "namespace": "", "name": "ip-10-31-75-171.eu-central-1.compute.internal", "reconcileID": "f96ab9db-d097-4811-a550-858e994c1727", "error": "no matches for kind \"PodDisruptionBudget\" in version \"policy/v1beta1\""}
2023-08-26T11:31:17.475Z INFO controller.provisioner.cloudprovider launched new instance {"commit": "06cb81f-dirty", "provisioner": "default", "launched-instance": "i-0ac9f6995b9188cc5", "hostname": "ip-10-31-75-206.eu-central-1.compute.internal", "type": "t3a.medium", "zone": "eu-central-1a", "capacity-type": "spot"}
2023-08-26T11:31:17.481Z INFO controller.provisioner.cloudprovider launched new instance {"commit": "06cb81f-dirty", "provisioner": "default", "launched-instance": "i-0cfe57b820e09ea4b", "hostname": "ip-10-31-75-216.eu-central-1.compute.internal", "type": "t3.large", "zone": "eu-central-1a", "capacity-type": "spot"}By using kubectl get nodes, you can observe the appearance of newly provisioned nodes:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-31-75-106.eu-central-1.compute.internal Unknown <none> 12s
ip-10-31-75-171.eu-central-1.compute.internal Unknown <none> 17s
ip-10-31-75-206.eu-central-1.compute.internal Unknown <none> 12s
ip-10-31-75-216.eu-central-1.compute.internal Unknown <none> 12s
ip-10-31-91-38.eu-central-1.compute.internal Ready <none> 34m v1.27.3-eks-a5565adAnd now they have transitioned to a Ready state:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-31-75-106.eu-central-1.compute.internal Ready <none> 44s v1.27.3-eks-a5565ad
ip-10-31-75-171.eu-central-1.compute.internal Ready <none> 49s v1.27.3-eks-a5565ad
ip-10-31-75-206.eu-central-1.compute.internal Ready <none> 44s v1.27.3-eks-a5565ad
ip-10-31-75-216.eu-central-1.compute.internal Ready <none> 44s v1.27.3-eks-a5565ad
ip-10-31-91-38.eu-central-1.compute.internal Ready <none> 34m v1.27.3-eks-a5565adFurthermore, all the pods that were previously in a Pending state have been provisioned on the newly created nodes:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
inflate-bbd699657-8gjkz 1/1 Running 0 3m1s
inflate-bbd699657-dzxgj 1/1 Running 0 3m1s
inflate-bbd699657-fjcpx 1/1 Running 0 3m1s
inflate-bbd699657-gqgb2 1/1 Running 0 3m1s
inflate-bbd699657-j97xs 1/1 Running 0 3m11s
inflate-bbd699657-lsj4x 1/1 Running 0 3m1s
inflate-bbd699657-mv9tp 1/1 Running 0 3m1s
inflate-bbd699657-nnzdj 1/1 Running 0 3m1s
inflate-bbd699657-sx6zh 1/1 Running 0 3m1s
inflate-bbd699657-v8ss2 1/1 Running 0 3m1s
inflate-bbd699657-w7m8n 1/1 Running 0 3m1s
inflate-bbd699657-xg7m8 1/1 Running 0 3m1sAfter adjusting the deployment scale to 1 using:
kubectl scale --replicas=1 deployment/inflateUpon checking the Karpenter logs, you will observe that the Nodes have become empty, and Karpenter has implemented a TTL of 30 seconds for managing these vacant nodes:
$ kubectl get logs -n karpenter karpenter-556f8d8f6b-55jg9
2023-08-26T11:35:39.580Z INFO controller.node added TTL to empty node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-171.eu-central-1.compute.internal"}
2023-08-26T11:35:39.584Z INFO controller.node added TTL to empty node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-106.eu-central-1.compute.internal"}
2023-08-26T11:35:39.605Z INFO controller.node added TTL to empty node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-216.eu-central-1.compute.internal"}
2023-08-26T11:35:39.612Z INFO controller.node added TTL to empty node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-206.eu-central-1.compute.internal"}Once the TTL period has elapsed, Karpenter will proceed to remove the nodes:
$ kubectl get logs -n karpenter karpenter-556f8d8f6b-55jg9
2023-08-26T11:36:18.552Z INFO controller.termination cordoned node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-171.eu-central-1.compute.internal"}
2023-08-26T11:36:18.568Z INFO controller.termination cordoned node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-206.eu-central-1.compute.internal"}
2023-08-26T11:36:18.590Z INFO controller.termination cordoned node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-106.eu-central-1.compute.internal"}
2023-08-26T11:36:18.614Z INFO controller.termination cordoned node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-216.eu-central-1.compute.internal"}
2023-08-26T11:36:18.760Z INFO controller.termination deleted node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-171.eu-central-1.compute.internal"}
2023-08-26T11:36:18.842Z INFO controller.termination deleted node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-206.eu-central-1.compute.internal"}
2023-08-26T11:36:18.901Z INFO controller.termination deleted node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-106.eu-central-1.compute.internal"}
2023-08-26T11:36:18.918Z INFO controller.termination deleted node {"commit": "06cb81f-dirty", "node": "ip-10-31-75-216.eu-central-1.compute.internal"}And voila! All the provisioned additional nodes have been successfully removed:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-31-91-38.eu-central-1.compute.internal Ready <none> 40m v1.27.3-eks-a5565adBefore continuing using Karpenter, remove the inflate deployment, so it wouldn’t use your precious resources.
Thank you! 🙏
