Run a scalable event-driven workflow with Amazon EKS, Keda and Karpenter.
How to take advantages of both worlds : Cloud Native Services and Kubernetes ? Here we’ll demonstrate how to compute on AWS managed kubernetes using events and managed services from AWS such as SQS.
I’m a huge kubernetes fan but when you are running a cluster on a cloud provider why not taking the best of both worlds ? You can have all the benefits of kubernetes and its community for your apps, and less maintenance by using cloud native services such as managed queues, managed storage, etc.
What is EKS and why not use AWS lambda for compute ?
EKS means Elastic Kubernetes Service. It’s the managed Kubernetes of AWS. You might have already a question, why should you run compute workloads on EKS rather than lambda or ECS ? Well, there is no good or bad answer to this question, because it all depends on your needs. Like we’re going to talk about Kubernetes here, I’ll give you 3 pros for k8s but I could give you more.
First pros I need to mention is portability. Kubernetes is available on most cloud providers, it can be deployed on-premises, and workloads are mostly portable. To balance a bit, certain Kubernetes patterns rely on service mesh in code like Istio for example. It means that your applications might require Kubernetes, and it is not portable to something other than Kubernetes. But like we said earlier, Kubernetes itself is portable.
Second pros, Kubernetes is a large ecosystem that has major support in every cloud, and a vast amount of community support. The CNCF landscape is an example of this vast ecosystem. In the EKS case, AWS provides add-ons and blueprints to help customers adopt some of these popular tools.
And last but not least, Kubernetes can solve issues you’re experiencing with AWS Lambda such as 15 minutes timeout for example but AWS ECS/fargate could solve it as well.
What is Keda ?
KEDA is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed.
KEDA is a single-purpose and lightweight component that can be added into any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication.
Keda provides you multiple scalers. Scalers will detect whether or not deployments should be active, scaled in or scaled out. It has various scalers such as some AWS events (SQS, cloudwatch, dynamodb, etc.), GCP events, Azure events, Loki, etc, etc.
Keda provides CRDs to define the autoscaling policy. The Keda operator will then use those policies to manage the scaling of our kubernetes objects.
What is Karpenter ?
Karpenter is a cluster autoscaler so based on pods needing disk, cpu and memory. It will add more nodes to a cluster to handle new workloads. Karpenter detects pending node, and based on AWS pricing and the list of EC2 instances you provided, it will choose the right instance size and add new machine(s) to your cluster.
Karpenter allows you to put quota on cpu and memory for your EKS cluster, so you don’t end up paying for resources you do not consume. Karpenter intercepts requests passed to Kubernetes admission controllers and takes action when demand does not match available resources. It means the operator listens to what is being deployed in the cluster and when the demand cannot be met by the current number of nodes, it will deploy new nodes. But the opposite works as well, when you have to much available unused resources, it will downscale your number of nodes.
Let’s get our hands DIRTY.
Usecase: Create a deployment where pods need 1 vCPU and 3GiB of memory. Pods need to be scaled up by keda when we post messages in an AWS SQS. If the current cluster can not handle the load, Karpenter comes into play and creates more nodes.
Prerequisites
Before deploying Keda or Karpenter we need to do some configurations. In order to have Keda and Karpenter interact with AWS in our case, they need to have credentials via IAM role for their service account. It provides the ability to manage credentials for your pods.
Your cluster has an OpenID Connect (OIDC) issuer URL associated with it. To use AWS Identity and Access Management (IAM) roles for service accounts, an IAM OIDC provider must exist for your cluster’s OIDC issuer URL. Here is the doc to create the IAM OIDC provider for your cluster.
This is the required policy for Keda to get AWS SQS messages and for Karpenter to work :
{
"Statement": [
## Karpenter
{
"Action": [
"ssm:GetParameter",
"ec2:DescribeImages",
"ec2:RunInstances",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeInstances",
"ec2:DescribeInstanceTypes",
"ec2:DescribeInstanceTypeOfferings",
"ec2:DescribeAvailabilityZones",
"ec2:DeleteLaunchTemplate",
"ec2:CreateTags",
"ec2:CreateLaunchTemplate",
"ec2:CreateFleet",
"ec2:DescribeSpotPriceHistory",
"pricing:GetProducts"
],
"Effect": "Allow",
"Resource": "*",
"Sid": "Karpenter"
},
## Karpenter
{
"Action": "ec2:TerminateInstances",
"Condition": {
"StringLike": {
"ec2:ResourceTag/Name": "*karpenter*"
}
},
"Effect": "Allow",
"Resource": "*",
"Sid": "ConditionalEC2Termination"
},
## Karpenter
{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/<POD_ROLE_NAME>",
"Sid": "PassNodeIAMRole"
},
## Karpenter
{
"Effect": "Allow",
"Action": "eks:DescribeCluster",
"Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}",
"Sid": "EKSClusterEndpointLookup"
},
## Karpenter
{
"Effect": "Allow",
"Action": "eks:DescribeCluster",
"Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}",
"Sid": "EKSClusterEndpointLookup"
},
## Keda
{
"Effect": "Allow",
"Action": "sqs:GetQueueAttributes",
"Resource": "<SQS_QUEUE_ARN>"
}
],
"Version": "2012-10-17"
}Once you role has been created, and your policy is attached, you need to modify the Trusted entities to associate the service account keda to the role.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "<OIDC_PROVIDER_ARN>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"<OIDC_PROVIDER_URL>:sub": "system:serviceaccount:<KUBERNETES_KEDA_NAMESPACE>:<SERVICE_ACCOUNT_NAME>"
}
}
}
]
}Deployment of Keda
We will use helm to deploy Keda. Run these 3 commands to get the values file :
helm repo add kedacore https://kedacore.github.io/charts
helm update repo
helm show values kedacore/keda > values.yamlBefore installing the release, we need to update the values. Here is what you need to add :
serviceAccount:
create: true
name: <SERVICE_ACCOUNT_NAME>
annotations:
eks.amazonaws.com/role-arn: <POD_ROLE_ARN>Now to deploy Keda, you need to run :
helm install keda kedacore/keda --values values.yaml --namespace <KUBERNETES_KEDA_NAMESPACE>To validate that we have Keda working well, you can run :
$ kubectl get pods -n keda
NAME READY STATUS RESTARTS AGE
keda-operator-<id> 1/1 Running 0 36s
keda-operator-metrics-apiserver-<id> 1/1 Running 0 36sDeployment of Karpenter
We will use kubectl to deploy Karpenter in this example, but you can use helm.
First you need to choose the version of Karpenter.
export KARPENTER_VERSION=v0.27.0Now, we’re going to generate a manifest to deploy Karpenter.
helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace karpenter \
--set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${NAME} \
--set settings.aws.clusterName=${CLUSTER_NAME} \
--set serviceAccount.annotations."eks.amazonaws.com/role-arn=<POD_ROLE_ARN>" \
--set controller.resources.requests.cpu=1 \
--set controller.resources.requests.memory=1Gi \
--set controller.resources.limits.cpu=1 \
--set controller.resources.limits.memory=1Gi > karpenter.yamlNow that our deployment is ready we can create the Karpenter namespace, create the CRDs, and then deploy the rest of the Karpenter resources.
kubectl create namespace karpenter
kubectl create -f \
https://raw.githubusercontent.com/aws/karpenter/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_provisioners.yaml
kubectl create -f \
https://raw.githubusercontent.com/aws/karpenter/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_awsnodetemplates.yaml
kubectl apply -f karpenter.yamlYou can check that Karpenter is up and running like with did with Keda. Now we need to deploy a default provisioner so Karpenter knows what types of nodes we want for unscheduled workloads. Below there is a small manifest example that allows Karpenter to create 2 more nodes.
cat <<EOF | kubectl create -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
labels:
karpenter.sh/demo: karpenter-demo
requirements:
- key: "node.kubernetes.io/instance-type"
operator: In
values: <list_of_instances_types> # [“t4g.medium”]
- key: "kubernetes.io/arch"
operator: In
values: ["amd64"]
- key: "karpenter.sh/capacity-type"
operator: In
values: ["on-demand", "spot"]
limits:
resources:
cpu: 4
memory: “8Gi”
provider:
launchTemplate: <launchTemplate> # toto-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
subnetSelector:
toto: toto
Tata: tata
EOFIn this manifest you can see the limits resources constrain that gives the maximum size of the cluster. It prevents Karpenter from creating new instances once the limit is reached. With t3.xlarge it allows Karpenter to create only 2 nodes in this configuration cause t4g.medium gives 2 vCPUs and 4GiB of memory. The launch template is mandatory because it’s the setup Karpenter will use to create new nodes.
How to test Karpenter then Keda.
- Karpenter
To test Karpenter let’s play with nodeAffinity to make it easier. We’re going to put a specific label (karpenter.sh/demo: karpenter-demo) on the Karpenter nodes. You shouldn’t have this label on your other node so if we put this as nodeAffinity on the deployment to directly trigger Kapenter to create new nodes because the pod won’t have any nodes to be created on.
kubectl create ns keda-test
kubectl config set-context --current --namespace=<KUBERNETES_KEDA_NAMESPACE>
kubectl create deployment nginx-deployment --image nginx --replicas=2 --requests=cpu=1,memory=3GiRunning those commands, it will deploy 2 pods with 1 vCPU and 3 GiB of memory so each pod will need on node and Karpenter should create to nodes. If you put 3 replicas, based on the current configuration of the provisioner Karpenter will create 2 nodes and let one pod pending.
2. Keda
With Keda we are going to scale the deployment replicas to zero because we’ll use an empty AWS SQS. Then we’re going to feed that queue to scale up and down the number of replicas.
To create a SQS, you can run :
aws sqs create-queue --queue-name keda-karpenter-demoWhen this queue has been created, we need to deploy the Keda scaled object and the trigger authentication.
cat <<EOF | kubectl create -f -
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: aws-sqs-queue-scaledobject
namespace: <KUBERNETES_KEDA_NAMESPACE>
spec:
scaleTargetRef:
name: nginx-deployment
minReplicaCount: 0 # We don't want pods if the queue is empty
maxReplicaCount: 2 # We don't want to have more than 5 replicas
pollingInterval: 10 # How frequently we should go for metrics (in seconds)
cooldownPeriod: 25 # How many seconds should we wait for downscale
triggers:
- type: aws-sqs-queue
authenticationRef:
name: keda-aws-credentials
metadata:
queueURL: https://sqs.<AWS_REGION>.amazonaws.com/<AWS_ACCOUNT_ID>/keda-karpenter-demo
queueLength: "1"
awsRegion: "<AWS_REGION>"
identityOwner: operator
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: keda-aws-credentials
namespace: <KUBERNETES_KEDA_NAMESPACE>
spec:
podIdentity:
provider: aws-eksAfter deploying the Keda configuration, like the queue is empty, the nginx deployment should be scaled in to 0 because the minReplicaCount is set to 0. Like we have two nodes without any resources running, Karpenter with downscale the number of nodes from 2 to 0.
The queue length set to 1, it means that with n messages, we’ll have n numbers of pods but with n < maxReplicaCount. In this case, with 2 messages in the queue, we’ll have 2 pods. With 3 messages in the queue, we’ll have 2 pods because of the Karpenter quota. Now let’s imagine, we have the queueLength set to 2, if we have 1 or 2 messages in the queue, we’ll have 1 pod, with 3–4 and more, we’ll have 2 pods because the maxReplicaCount is still set to 2.
To test this, you can send messages using :
for VARIABLE in 1..2
do
aws sqs send-message
--queue-url $(get-queue-url --queue-name keda-karpenter-demo)
--message-body "Keda and Karpenter demo"
doneThis should trigger the keda scaled object and create two pods. These two pods are going to be pending because of the cluster lack’s of space. Karpenter will intercept and detect those pending pods, it will create two nodes, and it’ll schedule those two pods on the new nodes.
Thanks for reading this article. Please do not hesitate to leave comments or questions if you have some.
