AWS EKS autoscaling by default (cluster|hpa)?

3 min readAug 20, 2021

You are using managed Kubernetes, configure your node group for a min, max, and desired instances, and expected that if need to spin up more nodes (of course respecting the max) will work out-of-the-box.

In fact, this is expected behavior at least for me because I running a lot of GKE clusters, and GKE dealing with that pretty well. And I surprise for EKS you need to create ASG, Policies, Roles, create service-account and deploy the cluster-autoscaling.

And if you expected to work with HPA (Horizontal Pod Autscaler) you will be disappointed as well, by default AWS does not provide metrics service, you need to install it manually.

If you are offering a service managed, that you already have the configs in the node group, why do you need set up other parts? Why AWS does not do it behind the scenes?

So that is one of the reasons I think the GCP is the best choice to embrace the culture of DevOps because they abstract most part the Ops for you.

Metrics Server

First, let's install the metrics server, it's pretty straightforward.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.0/components.yaml

And check it's running:

kubectl get apiservice v1beta1.metrics.k8s.io -o json | jq '.status'

Cluster Autoscale

If you create the EKS via web console or via cli eksctl they created for you the ASG (Auto Scaling Group) with the launch template, but you need to create policies and role, so let’s do this!

Identify what the ASG they created for you by CLI:

aws autoscaling \     
  describe-auto-scaling-groups \     
  --query "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name')]].[AutoScalingGroupName, MinSize, MaxSize,DesiredCapacity]" \     --output table

Or you can see in the Auto Scale Group section on web console https://console.aws.amazon.com/ec2autoscaling/home

Enabling IAM roles in the cluster

eksctl utils associate-iam-oidc-provider \
     --cluster eksworkshop-eksctl \
     --approve

Create policy

mkdir ~/environment/cluster-autoscaler  cat <<EoF > ~/environment/cluster-autoscaler/k8s-asg-policy.json 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeTags",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "ec2:DescribeLaunchTemplateVersions"
      ],
      "Resource": "*",
      "Effect": "Allow"
    }
  ]
} 
EoFaws iam create-policy   \
   --policy-name k8s-asg-policy \
   --policy-document file://~/environment/cluster-autoscaler/k8s-asg-policy.json

Create a role and kubernetes ServiceAccount

eksctl create iamserviceaccount \
     --name cluster-autoscaler \
     --namespace kube-system \
     --cluster ${CLUSTER_NAME} \
     --attach-policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/k8s-asg-policy" \
     --approve \
     --override-existing-serviceaccounts

Check the ServiceAccount

kubectl -n kube-system describe sa cluster-autoscaler

You can see the annotations you have the role

Now we need to deploy cluster-autoscale, for that we will use helm chart.

helm repo add autoscaler https://kubernetes.github.io/autoscalerhelm install cluster-autoscaler autoscaler/cluster-autoscaler \
--namespace kube-system \
--set 'autoDiscovery.clusterName'=<CLUSTER NAME> \
--set 'awsRegion'=<AWS_REGION>

But the trick for making it work is, your ASG and your EC2 instances MUST have 2 special TAGs

k8s.io/cluster-autoscaler/enabled = true
k8s.io/cluster-autoscaler/<YOUR_CLUSTER_NAME> = owned

First, goes to web console, ASG and edit the tags adding those.

Secondly, in the web console page for ASG the tag "Instance Management" you have a list of the EC2 instances, goes for each one edit the tags adding those listed above.

Finally, you can see the logs without the "no node group config"

AWS EKS autoscaling by default (cluster|hpa)?

Metrics Server

Cluster Autoscale

Credits

Written by Felipe Girotti