Rapid Auto-Scaling on EKS — Part 2
In Part 1 of this series, we examined how you can achieve Functional Node Readiness within 30 seconds on EKS. In this article, we will examine how we can improve the Bootstrapping and Provisioning time on EKS using Warm Pools.
Warm Pools for Auto Scaling
In April of 2021, AWS has launched a feature called “Warm Pools for Auto-Scaling Groups”, this allows you to keep stopped or running instances available as part of your scaling group, while maintaining a separate capacity pool. The more interesting use-case for EKS Clusters being Stopped instances, which outside of the EBS volume cost, are free. It allows you to replace the Provisioning time by the time it takes your OS to boot up, since the instance is already provisioned, just stopped, and this can be additional time we can save from the scaling process.
Unfortunately, when we looked into integrating this to instance-manager which is our operator for provisioning scaling groups — we noticed the docs mention “Warm pools currently can’t be used with ECS, EKS, and self-managed Kubernetes”.
But we wanted to know why, exactly why.
“RTFM” — No.
Nothing wrong with disregarding documentation and trying something for yourself, don’t get me wrong, we also reached out to AWS to ask “Why exactly”, but while waiting for their answer we just tried using it, and we quickly found the answer.
In order to reach the lifecycle state “Warmed:Stopped”, you first have to go through a few other states, but essentially the EC2 instance is started as it normally is, runs it’s user-data script, and once user-data has finished it’s execution, the instance is stopped. The problem is that between user-data’s completion and shutting down the instance, there is a time window where the node actually might join the cluster, and due to our previous improvements can quickly become ready and actually start scheduling workloads, only to be shut down few moments later.
We noticed this might leave a NotReady node in the cluster, or that eventually Kubernetes removes the nodes — but it still seemed like a bad thing to put in production, we had to find a way to skip bootstrapping the node if it was simply warming up, and only allow bootstrapping when the instance transitions into the capacity pool due to actual scaling.
While we figured out what the problem is, AWS changed their docs to reflect the same reasoning, at the time of writing this, it reads “If you try using warm pools with Amazon Elastic Container Service (Amazon ECS) or Elastic Kubernetes Service (Amazon EKS) managed node groups, there is a chance that these services will schedule jobs on an instance before it reaches the warm pool.”
The Workaround
So until AWS rolls out some sort of native support, we wanted to find a workaround for this issue, so that we can enjoy the benefits of warm pools and possibly remove extra precious seconds from our scale time.
We needed to basically skip the bootstrapping entirely, in case the instance was getting warmed up, and then make sure that the next boot-up, when actual scaling occurs, user-data runs again, this time, bootstrapping the node to the cluster.
So, with a bit of tinkering, we were able to modify our user-data script to do exactly that, in the case of Amazon Linux 2 it looked like this:
And in the case of Windows:
This worked perfectly! The downside was that we introduced additional dependencies on the AMI for users who want to use warm pools with instance-manager — in the case of Amazon Linux 2, the AMI needs to have both jq and awscli and in addition to that the IAM role associated with the instance, must have access to DescribeAutoscalingInstances so that it can check it’s own lifecycle state and see whether it contains the “Warmed” prefix. The same is relevant for Windows only instead of those packages we require the Get-ASAutoScalingInstance cmdlet installed.
The API for using warm pools with instance-manager is also very simple, just add WarmPoolSpec your InstanceGroup EKSSpec and define the min/max ranges, you can use -1 to reflect the scaling group’s max, however it probably doesn't make sense to run more than a few warm instances for even large scaling groups.
apiVersion: instancemgr.keikoproj.io/v1alpha1
kind: InstanceGroup
metadata:
name: hello-world
namespace: instance-manager
spec:
provisioner: eks
eks:
maxSize: 6
minSize: 3
warmPool:
maxSize: -1
minSize: 0
configuration:
< .. additional configuration .. >The Results
After running a few very basic tests, we finally got the results.
On Amazon Linux 2, Provisioning time went from around ~2 minutes, to ~1 minute.
On Windows, Provisioning time went from ~7 minutes to ~3 minutes.
Around a 50% improvement — not bad!
So given all of these changes we made, with warm pools, and cluster configuration changes, where did we start and where did we end up?
Total time from a pending pod to running pod (with auto-scaling).
Before: ~4 to 5.5 minutes (depends on race conditions discussed in part 1)
After: 90 seconds average
If you want to use warm pools with EKS, now you can - if you integrate instance-manager. It answers our use cases for platform management and is deployed on each of our 182 EKS clusters, here is a previous article we published recently about the specifics of instance-manager and the use-cases it answers.
