How to update EKS node types?

While using Leverage, creating an EKS cluster becomes a straightforward process. To guide you through this, please refer to this link. This scenario involves upgrading the node instance types.

Workloads considerations

Ideal scenario

In an optimal situation, all workloads should be scalable. This implies that all workloads managed through ArgoCD Application’s Rollouts, the default in binbash Leverage, should be capable of upscaling. This means creating multiple pods for each ArgoCD Application to handle increased demand or load.

Non-ideal solution

However, in instances where certain applications can only accommodate a single instance (i.e., one pod), scaling becomes challenging. In such cases, there may be downtime as the existing pod needs to be terminated, and a new one must be created on the new nodes. This limitation poses challenges in maintaining continuous availability during scaling activities for these specific applications.

Steps

Create a new node group

Go to the `cluster` sublayer under the EKS layer at `<leverage-project-name>/<account-name>/<region>/k8s-eks/cluster` . (e.g. here)

Edit the file `eks-managed-nodes.tf`. (e.g. here)

Under the EKS module:

module "cluster" {
 source = "github.com/binbashar/terraform-aws-eks.git?ref=v18.24.1"

there is a node group definition, e.g.

eks_managed_node_groups = {
 on-demand-t3 = {
 min_size = 2
 max_size = 10
 desired_size = 6
 capacity_type = "ON_DEMAND"
 instance_types = ["t3.large"]
 }
 }

(note this is an ON_DEMAND type, it also can be SPOT)

Add a new node group, e.g.:

eks_managed_node_groups = {
 on-demand-t3 = {
 min_size = 2
 max_size = 10
 desired_size = 6
 capacity_type = "ON_DEMAND"
 instance_types = ["t3.large"]
 }
 on-demand-t3xlarge = {
 min_size = 1
 max_size = 6
 desired_size = 3
 capacity_type = "ON_DEMAND"
 instance_types = ["t3.xlarge"]
 }
 }

(adjust the values as per your needs, here I’m considering the xlarge instance has the double of RAM and CPU than large)

Apply the layer and wait for the group to be created.

Cordon the old group

Once the new group is created and the new nodes are in place, the old group has to be cordoned to avoid scheduling in those nodes.

First, under the `cluster` layer, log into the EKS cluster:

❯ leverage kubectl configure

Get the node groups:

❯ leverage aws - profile <project-short-name>-<account-name>-devops - region <region> eks list-nodegroups - cluster-name <cluster-name>
{
 "nodegroups": [
 "on-demand-t3–20220803130748876100000008",
 "on-demand-t3xlarge-20231130160430016700000001",
 ]
}

Note here there are two groups, the old one and the new one.

Now cordon the nodes in the old node group:

❯ for n in $(leverage kubectl get nodes - selector eks.amazonaws.com/nodegroup=on-demand-t3–20220803130748876100000008 | awk 'FNR>5 {print $1}'); do leverage kubectl cordon $n; done

It can be checked the old nodes are cordoned with this command:

❯ leverage kubectl get nodes - selector eks.amazonaws.com/nodegroup=on-demand-t3–20220803130748876100000008
[16:58:55] INFO Attempting to get temporary credentials for apps-dev account.
[16:58:56] INFO Using already configured temporary credentials.
[16:58:56] INFO Attempting to get temporary credentials for shared account.
[16:58:57] INFO Using already configured temporary credentials.
NAME STATUS ROLES AGE VERSION
ip-10–0–13–118.ec2.internal Ready,SchedulingDisabled <none> 13d v1.24.17-eks-43840fb
ip-10–0–40–140.ec2.internal Ready,SchedulingDisabled <none> 20d v1.24.17-eks-43840fb
ip-10–0–52–96.ec2.internal Ready,SchedulingDisabled <none> 20d v1.24.17-eks-43840fb
ip-10–0–6–43.ec2.internal Ready,SchedulingDisabled <none> 13d v1.24.17-eks-43840fb

Names can vary, but note the SchedulingDisable status, this means nodes are cordoned, i.e. no pod will be scheduled in these nodes.

Avoid old group to scale up

To prevent the old group from scaling-up, do the following.

Check the amount of old group nodes. It can be seen in the previous step (checking cordoned nodes) there are 4 nodes.

So, edit the file again and set the max and desired size to 4.

eks_managed_node_groups = {
 on-demand-t3 = {
 min_size = 2
 max_size = 4
 desired_size = 4
 capacity_type = "ON_DEMAND"
 instance_types = ["t3.large"]
 }
 on-demand-t3xlarge = {
 min_size = 1
 max_size = 6
 desired_size = 3
 capacity_type = "ON_DEMAND"
 instance_types = ["t3.xlarge"]
 }
 }

Apply the layer.

Migrate Applications

Escalable Apps
If Applications can scale up (i.e. managing more than one pod), do the following for each app.

- Go to the ArgoCD web console
- Pick the Application
- Turn AutoSync off
- Scale up (set the double amount of pods, i.e. if there are 2 pods, set the new value to 4)
— if no HPA
— in the Rollout definition change the Replicas value
— if HPA
— in HPA set the min amount of replicas
- Wait until new pods are created

Pods being created in the new nodes can be checked with the following command:

for n in $(leverage kubectl get nodes - selector "eks.amazonaws.com/nodegroup!=on-demand-t3–20220803130748876100000008" | awk 'FNR>5 {print $1}'); do leverage kubectl get pods - all-namespaces -o wide - field-selector spec.nodeName=$n; done

- Once the new pods are created, old pods can be killed.
- Once all the pods for an Application are scheduled in the new nodes the Replicas values can be set back to the original value
- Turn on auto sync

No escalable Apps
If app can not be escalated:

- Set a Maintenance Window for the process
- During the MW do the following:
— Go to ArgoCD Web Console
— Pick the Application
— Kill the existing pod
— Wait until the new pod is created

Once all the non-escalabe-apps are done you can continue with the next step.

Drain nodes

It is recommended to do this node by node manually to keep control over the process.

For each node in the old node group run:

❯ leverage kubectl drain node-name --ignore-daemonsets

During this process, if this mesage is shown “cannot delete Pods with local storage”….
This probably is due to “argocd/argocd-image-updater” or “kube-system/coredns” in the node using local (emptyDir) storage.
In this case it is safe to add the flag:

--delete-emptydir-data

Delete the old node group

Edit the file one more time, delete the old node group so it looks like this:

on-demand-t3xlarge = {
 min_size = 1
 max_size = 6
 desired_size = 3
 capacity_type = "ON_DEMAND"
 instance_types = ["t3.xlarge"]
 }
 }

Apply the layer.

Conclusion

This process is aimed to update the EKS nodes type with no downtime (or minimal and controlled downtime).

Note it is recommended to have a node group per AZ so in a given moment there is at least one node per AZ, this is to enforce HA.
So this can be set in the file:

on-demand-t3xlarge-a = {
 min_size = 2
 max_size = 10
 desired_size = 2
 capacity_type = "ON_DEMAND"
 instance_types = ["t3.xlarge"]
 subnet_ids = [data.terraform_remote_state.eks-vpc.outputs.private_subnets[0]]
 placement = {
 availability_zone = data.terraform_remote_state.eks-vpc.outputs.availability_zones[0]
 }
 }
 on-demand-t3xlarge-b = {
 min_size = 2
 max_size = 10
 desired_size = 2
 capacity_type = "ON_DEMAND"
 instance_types = ["t3.xlarge"]
 subnet_ids = [data.terraform_remote_state.eks-vpc.outputs.private_subnets[1]]
 placement = {
 availability_zone = data.terraform_remote_state.eks-vpc.outputs.availability_zones[1]
 }
 }

Note here we are using Terraform remote states from the EKS `network` sublayer (e.g. here), so something like this is needed in the `config.tf` file:

data "terraform_remote_state" "eks-vpc" {
 backend = "s3"
 config = {
 region = var.region
 profile = var.profile
 bucket = var.bucket
 key = "<account-name>/k8s-eks/network/terraform.tfstate"
 }
}