On Amazon EKS and cluster add-ons

The Amazon Elastic Kubernetes Service, or Amazon EKS for short, is a great place to run your Kubernetes workloads. Amazon EKS is a managed service that makes it easier to run Kubernetes on AWS without installing and operating the Kubernetes control plane components yourself, which is excellent as you can leave the lifecycle management of the control plane to AWS. However, many other things are required to operationalise an Amazon EKS cluster in a way that makes it ready to host application workloads… One of the questions I get frequently asked about is how one might best go about running cluster-addons on Amazon EKS.

Besides being very considerate about which cluster add-ons to choose from the ever-growing landscape of options, anyone who prepares for running Amazon EKS at scale and in production typically needs to decide how to deploy and operate their preferred set of add-ons. There are several options available on how to achieve this. Equally, there may be cases where these options can be combined to optimise for a particular situation. One way to start reasoning about this is to pin down some of the well-known options:

EKS managed add-ons: This is an excellent option if you prefer to have AWS do the lifecycle management for you. You can, for example, deploy and upgrade Amazon EKS add-ons via the AWS CLI, eksctl cli, and the AWS Management Console. These add-ons include the latest security patches and bug fixes and are validated by the EKS services team to work with Amazon EKS. The Amazon EKS team is increasing the number of available EKS managed add-ons over time, but you’re unlikely to find all the add-ons you’d need there.

Add-ons via EKS Blueprints: EKS Blueprints are a collection of Infrastructure as Code (IaC) modules that help us configure and deploy cluster add-ons consistently across Amazon EKS clusters, accounts and regions. The nice thing here is that this framework lets you choose and use add-ons from the range of available Amazon EKS managed add-ons and popular open-source add-ons. You’d need to be friendly with either AWS Cloud Development Kit (AWS CDK) or HashiCorp Terraform to use this option.

Self-managed add-ons: Here, the expectation would be that “you really know what you’re doing” and that you probably need a lot of control over how the cluster add-ons ought to be handled. This opens up some very interesting considerations and probably offers the most significant degree of flexibility and control, albeit at the cost of needing to make a set of decisions yourself.

There are already a good set of helper docs and Getting Started Guides available, including EKS Blueprints for CDK and EKS Blueprints for Terraform. Running self-managed add-ons on your Amazon EKS clusters is also a great option: This blog takes a closer look at how to make this work.

To start with, we could run our self-managed cluster add-ons directly on the data plane worker nodes and let the Kubernetes scheduler place add-ons onto any node. In this way, the add-ons run alongside user application workloads and will all share the same pool of worker nodes.

Remember, we cannot deploy anything into the Kubernetes master nodes, as Amazon EKS is a managed control plane. Therefore, we cannot place any workloads on them, let alone see nor access these nodes directly. And for good reason :-)

The worker nodes would typically be EC2 instances that are bootstrapped within node groups, such as Managed Node Groups or Self-Managed Nodes. These node groups are then backed by EC2 Autoscaling Groups, or ASGs for short, so we can automatically or manually scale our sets of worker node machines. Typically, you’d define many such Node Groups as your needs for differentiated instance types grow and change over time. The following diagram illustrates this whilst also showing the options of defining Node Groups across and within Availability Zones.

Cluster add-ons and application workloads share the data plane

Now, we can also choose to introduce worker node tiers to reserve a set of worker nodes that exclusively host our cluster add-ons. This would separate our add-ons from user application workloads. A way to achieve that would be to provision a dedicated Node Group with taints so that the Kubernetes scheduler places our add-ons that have the matching tolerations onto the desired node tier. This enables us to separately manage the lifecycle of the add-on tier and control for the criticality of the add-on pods. In this way, we reduce the risk of things such as the “noisy-neighbour-effect” of application workloads, which may impact the stability of our add-ons.

Cluster add-ons running on a dedicated Node Group

A third alternative would be to run our cluster add-ons on serverless compute, i.e. AWS Fargate. With AWS Fargate being a serverless compute engine, we’d not be provisioning and operating EC2 instances ourselves nor would we have things as Node Groups or ASGs. This opens up the idea of running our cluster add-ons on a serverless compute tier and then, if we so decide, use autoscaler add-ons such as Karpenter or Cluster Autoscaler to scale our EC2-based Node Groups.

Cluster add-ons on AWS Fargate serverless

In fact, for those who can, AWS Fargate could also be a great option to run your application workloads on. Before you make that leap, there are some subtleties to observe, but I’ve had some great experience running Cluster Add-ons on AWS Fargate.

Let’s do it…

For those on a tight time budget: The TL;DR of the following sections is to show how cluster add-ons - in our example we use Karpenter - can be run on AWS Fargate serverless compute so that we avoid having to deal with Node Groups.

I’ve become friendly with eksctl over time, hence it appears in the snippets, but you can, of course, use your own favourite sets of CLIs to get the job done.

The premise of these sections is to illustrate the underpinning mechanics of running add-ons on AWS Fargate in a sandbox environment… so we won’t end up with something that is production ready.

Creating our Amazon EKS cluster

First, let’s start with provisioning a basic Amazon EKS cluster control plane. Set your variables and then use eksctl, which can look something like this:

$ cat << EOF > cluster-init.yaml
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: $CLUSTER_NAME
  region: ${AWS_REGION}
  version: "1.21"
  tags:
    karpenter.sh/discovery: ${CLUSTER_NAME}
availabilityZones: ["${AZS[0]}", "${AZS[1]}", "${AZS[2]}"]
vpc:
  clusterEndpoints:
    publicAccess: true
    privateAccess: false
iam:
  withOIDC: true
fargateProfiles:
  - name: karpenter
    selectors:
    - namespace: kube-system
    - namespace: karpenter
EOF
$ eksctl create cluster -f cluster-init.yaml

Eksctl is opinionated and will add default settings for things we did not explicitly define and pass them into the Amazon EKS provisioning stacks on our behalf. Eksctl will, amongst other things, update your ~/.kube/config file, which is very convenient as you’re all set to access the cluster with kubectl.

Creating our AWS Fargate profile

We’ve already passed in our AWS Fargate profiles creation flags into eksctl at cluster creation time, but we could also use eksctl create fargateprofile to create Fargate profiles separately. The nice thing here is that eksctl also creates the required pod execution role under the hood, if one doesn’t already exist. This role ensures the Fargate workloads acquire the permissions to interact with AWS APIs so that they can do things such as register their kublets with the Amazon EKS cluster and access Amazon ECR image repositories.

If you create the cluster with the eksctl --fargate option as shown, then the AWS Fargate profile is created with selectors for all pods in the kube-system and karpenter namespaces.

Amazon EKS and AWS Fargate work nicely together, as AWS Fargate profiles can be associated with a Kubernetes namespace. All workloads scoped into such a namespace are then scheduled into AWS Fargate. We could also achieve similar scheduling behaviour with labels.

With the Amazon EKS cluster and AWS Fargate profiles in place, we’re ready to prepare the deployment of one of my favourite cluster add-ons: Karpenter. Karpenter is one of those add-ons requiring additional IAM permissions as it interacts with AWS EC2 APIs to operate correctly and will provision EC2 instances in your account. Hence it is a valuable add-on example to choose, as you’re likely to end up using many add-ons that will require AWS API permissions, as they are wonderfully useful.

To show how this is done, we now broadly follow the steps set out in the Karpenter documents.

IAM Permissions for add-ons

We start by using a quick helper AWS CloudFormation stack from the Karpenter repo that will create the KarpenterNode IAM Role for us and attach a set of AWS managed IAM policies to it. Why? Any EC2 instances launched by Karpenter must run with the InstanceProfile that grants the permissions necessary to run containers and configure networking.

To do this, we set some variables first that we’ll be using as part of the code snippets to help us along the way. Notice the --profile flag, as you’ll probably want to refer to the AWS CLI profile name in the ~/.aws/credentials file that contains your AWS account details.

$ export KARPENTER_VERSION=v0.19.0
$ export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text --profile $MY_PROFILE_NAME)"
$ export KARPENTER_IAM_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter" 
$ export CLUSTER_ENDPOINT="$(aws eks describe-cluster --name
${CLUSTER_NAME} --query "cluster.endpoint" --output text --profile $MY_PROFILE_NAME)"

Then we launch the helper stack, which creates an IAM policy named KarpenterControllerPolicy, an IAM instance profile named KarpenterNodeInstanceProfile, and an IAM role named KarpenterNodeRole.

$ TEMPOUT=$(mktemp)
$ curl -fsSL https://karpenter.sh/"${KARPENTER_VERSION}"/getting-started/getting-started-with-eksctl/cloudformation.yaml  > $TEMPOUT \
&& aws cloudformation deploy --profile $MY_PROFILE_NAME \
  --stack-name "Karpenter-${CLUSTER_NAME}" \
  --template-file "${TEMPOUT}" \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides "ClusterName=${CLUSTER_NAME}"

Now we do an aws-auth configmap update on our Amazon EKS cluster via eksctl iamidentitymapping so that our KarpenterNode IAM Role acquires the Kubernetes RBAC permissions we need for it to bootstrap and join new nodes into the cluster.

$ eksctl create iamidentitymapping --profile $MY_PROFILE_NAME \
  --username system:node:{{EC2PrivateDNSName}} \
  --cluster "${CLUSTER_NAME}" \
  --arn  "arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}" \
  --group system:bootstrappers \
  --group system:nodes

With all that done, we need a supplementary IAM Role for the Karpenter add-on itself, i.e. the Controller. We use eksctl to create another IAM Role and attach a policy to it. This IAM Role will then be referenced via an annotation in the Karpenter Kubernetes ServiceAccount in a later step when we install the Karpenter add-on. Assigning fine-grain IAM permissions at a Kubernetes ServiceAccount level for pods is known as IAM Roles for Service Accounts, or IRSA for short. This is much better than using the Amazon EC2 Instance Profile Role.

Here we go:

$ eksctl create iamserviceaccount --profile=$MY_PROFILE_NAME \
  --cluster "${CLUSTER_NAME}" --name karpenter --namespace karpenter \
  --role-name "${CLUSTER_NAME}-karpenter" \
  --attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" \
  --role-only \
  --approve

Eksctl now creates and applies a CloudFormation stack to provision the IAM role.

Installing Karpenter on AWS Faregate

And now that we’re all done with IAM Roles and IRSA, we’re ready to install our Karpenter add-on.

There are many options on how to deploy add-ons and workloads in general, ranging from the unspeakable CLI command-line, raw manifests, kustomizations, and helm charts, all the way to perhaps more elaborate approaches such as GitOps that help with handling lifecycle management of workloads. For this blog, let’s go ahead and use the Karpenter helm chart. You’ll want to get the helm CLI for this, in case you don’t already have it installed on whatever environment you are on.

Then we add the Karpenter helm repo and install the chart.

$ export KARPENTER_IRSA_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter"
$ helm repo add karpenter https://charts.karpenter.sh/ 
$ helm repo update 
$ helm show values karpenter/karpenter
$ helm upgrade --install --namespace karpenter --create-namespace \
  karpenter karpenter/karpenter \
  --version ${KARPENTER_VERSION} \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IRSA_ROLE_ARN} \
  --set clusterName=${CLUSTER_NAME} \
  --set clusterEndpoint=${CLUSTER_ENDPOINT} \
  --set aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
  --wait

Give it a moment to install, as AWS Fargate needs to create micro-VMs for us on which to place the pods.

Now we can define a Provisioner custom resource that Karpenter provides. What is it? The Provisioner allows us to define instructions for Karpenter on how we want the scaling to behave. We define a default Provisioner with some basic properties. More on that in a separate blog.

$ cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand"]
  limits:
    resources:
      cpu: 100
  provider:
    subnetSelector:
      karpenter.sh/discovery: ${CLUSTER_NAME}
    securityGroupSelector:
      karpenter.sh/discovery: ${CLUSTER_NAME}
  ttlSecondsAfterEmpty: 30
EOF

With all that done, Karpenter is ready to begin provisioning EC2-based worker nodes in our account.

Notice how we declared the karpenter.sh/discoverytag in the Provisioner yaml and also at the beginning during cluster creation. Karpenter uses these tags to auto-discover subnets and securitygroups, without which it cannot provision EC2 instances.

We can deploy a new workload into the cluster, e.g. the default namespace, and watch the behaviour. We’ll see new pods go into the pending state as no worker nodes are found to place them on. The Karpenter controller will detect this and provision a new EC2-based worker node that best fits our requirement statements in the Provisioner and schedule the pods.

Happy days :-)

Conclusion:

We saw that opting for EKS managed add-ons can be very convenient. However, this would leave us without many add-ons we know we need but are not yet available under the managed add-on framework. Therefore we need to extend our options and look towards e.g. Managed Node Groups and EKS Blueprints to help us host, deploy and operate the remaining add-ons we need. And if you are prepared to go the extra mile with handling IAM and IRSA, or you need additional control, then I would opt for running supplementary cluster add-ons very elegantly and efficiently on AWS Fargate.