How to update EKS node types?
While using Leverage, creating an EKS cluster becomes a straightforward process. To guide you through this, please refer to this link. This scenario involves upgrading the node instance types.
Workloads considerations
Ideal scenario
In an optimal situation, all workloads should be scalable. This implies that all workloads managed through ArgoCD Application’s Rollouts, the default in binbash Leverage, should be capable of upscaling. This means creating multiple pods for each ArgoCD Application to handle increased demand or load.
Non-ideal solution
However, in instances where certain applications can only accommodate a single instance (i.e., one pod), scaling becomes challenging. In such cases, there may be downtime as the existing pod needs to be terminated, and a new one must be created on the new nodes. This limitation poses challenges in maintaining continuous availability during scaling activities for these specific applications.
Steps
Create a new node group
Go to the `cluster` sublayer under the EKS layer at `<leverage-project-name>/<account-name>/<region>/k8s-eks/cluster` . (e.g. here)
Edit the file `eks-managed-nodes.tf`. (e.g. here)
Under the EKS module:
module "cluster" {
source = "github.com/binbashar/terraform-aws-eks.git?ref=v18.24.1"
there is a node group definition, e.g.
eks_managed_node_groups = {
on-demand-t3 = {
min_size = 2
max_size = 10
desired_size = 6
capacity_type = "ON_DEMAND"
instance_types = ["t3.large"]
}
}
(note this is an ON_DEMAND type, it also can be SPOT)
Add a new node group, e.g.:
eks_managed_node_groups = {
on-demand-t3 = {
min_size = 2
max_size = 10
desired_size = 6
capacity_type = "ON_DEMAND"
instance_types = ["t3.large"]
}
on-demand-t3xlarge = {
min_size = 1
max_size = 6
desired_size = 3
capacity_type = "ON_DEMAND"
instance_types = ["t3.xlarge"]
}
}
(adjust the values as per your needs, here I’m considering the xlarge instance has the double of RAM and CPU than large)
Apply the layer and wait for the group to be created.
Cordon the old group
Once the new group is created and the new nodes are in place, the old group has to be cordoned to avoid scheduling in those nodes.
First, under the `cluster` layer, log into the EKS cluster:
❯ leverage kubectl configure
Get the node groups:
❯ leverage aws - profile <project-short-name>-<account-name>-devops - region <region> eks list-nodegroups - cluster-name <cluster-name>
{
"nodegroups": [
"on-demand-t3–20220803130748876100000008",
"on-demand-t3xlarge-20231130160430016700000001",
]
}
Note here there are two groups, the old one and the new one.
Now cordon the nodes in the old node group:
❯ for n in $(leverage kubectl get nodes - selector eks.amazonaws.com/nodegroup=on-demand-t3–20220803130748876100000008 | awk 'FNR>5 {print $1}'); do leverage kubectl cordon $n; done
It can be checked the old nodes are cordoned with this command:
❯ leverage kubectl get nodes - selector eks.amazonaws.com/nodegroup=on-demand-t3–20220803130748876100000008
[16:58:55] INFO Attempting to get temporary credentials for apps-dev account.
[16:58:56] INFO Using already configured temporary credentials.
[16:58:56] INFO Attempting to get temporary credentials for shared account.
[16:58:57] INFO Using already configured temporary credentials.
NAME STATUS ROLES AGE VERSION
ip-10–0–13–118.ec2.internal Ready,SchedulingDisabled <none> 13d v1.24.17-eks-43840fb
ip-10–0–40–140.ec2.internal Ready,SchedulingDisabled <none> 20d v1.24.17-eks-43840fb
ip-10–0–52–96.ec2.internal Ready,SchedulingDisabled <none> 20d v1.24.17-eks-43840fb
ip-10–0–6–43.ec2.internal Ready,SchedulingDisabled <none> 13d v1.24.17-eks-43840fb
Names can vary, but note the SchedulingDisable status, this means nodes are cordoned, i.e. no pod will be scheduled in these nodes.
Avoid old group to scale up
To prevent the old group from scaling-up, do the following.
Check the amount of old group nodes. It can be seen in the previous step (checking cordoned nodes) there are 4 nodes.
So, edit the file again and set the max and desired size to 4.
eks_managed_node_groups = {
on-demand-t3 = {
min_size = 2
max_size = 4
desired_size = 4
capacity_type = "ON_DEMAND"
instance_types = ["t3.large"]
}
on-demand-t3xlarge = {
min_size = 1
max_size = 6
desired_size = 3
capacity_type = "ON_DEMAND"
instance_types = ["t3.xlarge"]
}
}
Apply the layer.
Migrate Applications
Escalable Apps
If Applications can scale up (i.e. managing more than one pod), do the following for each app.
- Go to the ArgoCD web console
- Pick the Application
- Turn AutoSync off
- Scale up (set the double amount of pods, i.e. if there are 2 pods, set the new value to 4)
— if no HPA
— in the Rollout definition change the Replicas value
— if HPA
— in HPA set the min amount of replicas
- Wait until new pods are created
Pods being created in the new nodes can be checked with the following command:
for n in $(leverage kubectl get nodes - selector "eks.amazonaws.com/nodegroup!=on-demand-t3–20220803130748876100000008" | awk 'FNR>5 {print $1}'); do leverage kubectl get pods - all-namespaces -o wide - field-selector spec.nodeName=$n; done
- Once the new pods are created, old pods can be killed.
- Once all the pods for an Application are scheduled in the new nodes the Replicas values can be set back to the original value
- Turn on auto sync
No escalable Apps
If app can not be escalated:
- Set a Maintenance Window for the process
- During the MW do the following:
— Go to ArgoCD Web Console
— Pick the Application
— Kill the existing pod
— Wait until the new pod is created
Once all the non-escalabe-apps are done you can continue with the next step.
Drain nodes
It is recommended to do this node by node manually to keep control over the process.
For each node in the old node group run:
❯ leverage kubectl drain node-name --ignore-daemonsets
During this process, if this mesage is shown “cannot delete Pods with local storage”….
This probably is due to “argocd/argocd-image-updater” or “kube-system/coredns” in the node using local (emptyDir) storage.
In this case it is safe to add the flag:
--delete-emptydir-data
Delete the old node group
Edit the file one more time, delete the old node group so it looks like this:
on-demand-t3xlarge = {
min_size = 1
max_size = 6
desired_size = 3
capacity_type = "ON_DEMAND"
instance_types = ["t3.xlarge"]
}
}
Apply the layer.
Conclusion
This process is aimed to update the EKS nodes type with no downtime (or minimal and controlled downtime).
Note it is recommended to have a node group per AZ so in a given moment there is at least one node per AZ, this is to enforce HA.
So this can be set in the file:
on-demand-t3xlarge-a = {
min_size = 2
max_size = 10
desired_size = 2
capacity_type = "ON_DEMAND"
instance_types = ["t3.xlarge"]
subnet_ids = [data.terraform_remote_state.eks-vpc.outputs.private_subnets[0]]
placement = {
availability_zone = data.terraform_remote_state.eks-vpc.outputs.availability_zones[0]
}
}
on-demand-t3xlarge-b = {
min_size = 2
max_size = 10
desired_size = 2
capacity_type = "ON_DEMAND"
instance_types = ["t3.xlarge"]
subnet_ids = [data.terraform_remote_state.eks-vpc.outputs.private_subnets[1]]
placement = {
availability_zone = data.terraform_remote_state.eks-vpc.outputs.availability_zones[1]
}
}
Note here we are using Terraform remote states from the EKS `network` sublayer (e.g. here), so something like this is needed in the `config.tf` file:
data "terraform_remote_state" "eks-vpc" {
backend = "s3"
config = {
region = var.region
profile = var.profile
bucket = var.bucket
key = "<account-name>/k8s-eks/network/terraform.tfstate"
}
}