The Trio — EKS, Karpenter and Fargate
AWS has continued to push the business case as to why to adopt Kubernetes. The continued contributions like IRSA (IAM Roles for Service Accounts), CSI Drivers, AWS IAM Authenticator and more recently Karpenter, has continued to make the integration between Kubernetes and the AWS ecosystem much more cohesive.
Karpenter is an Open Source tool built by AWS to do more efficient autoscaling by provisioning instances by themselves based on the current need of the cluster and not by using autoscaling groups like the default cluster autoscaler will do. This has made node provisioning faster but also much more cost efficient. Karpenter has also added features to integrate with AWS Health to listen for node health events, spot interruptions, to deal with things before they happen. There is also configuration you can setup to set TTL for nodes to ensure nodes are not long lived and to be able to shuffle nodes around for cost savings.
You may be asking yourself why not just use Fargate for my entire cluster, and there is a number of reasons in my mind why Fargate may not be the best choice for your entire cluster, however this really depends on your use case. Fargate doesn’t support Daemonsets, if you use things like Fluentd, Cilium, etc; they are not going to work on Fargate nodes, one workaround is to run them as sidecars. Sidecars at scale can be messy so this is something you’ll want to be careful of. If you do need access to the host for things like eBPF, a service mesh, etc; Fargate will not allow those systems to get access to the parts of the host it needs to operate. Take a look at some of the other limitations of Fargate first, and ensure that your use case will or won’t work and be cost efficient before making a decision.
Karpenter runs on normal instances and not on the control plane, this means the way you deploy Karpeneter has to be resilient. What is the best practice is to deploy it on to either Fargate nodes, or ASG based EC2 instances, and what is a better use case for Fargate then workloads that don’t need to be integrated with a service mesh, or some of the other tooling you’d use on an EC2 instance. Fargate will be reliable, and for each pod has a 99.5% uptime per month, which means at most you have a four hour outage per month. You should ensure you have a Disaster recovery plan if Fargate fails to work or does have an outage however for the majority of the time Fargate works well and as expected.
There are even more ways to optimize compute with tools like Kube Descheuler. One of those being a Kubernetes SIG (special interest group) project called Kube Descheduler. This tool allows you to setup policies on when it should try and optimize (desired amount of CPU/Memory, if pod has high amount of restarts, etc;).
More Resources: