Everything about Kubernetes : Cluster Maintenance

Kubernetes maintenance mode is a state where a node or cluster is taken offline for updates, repairs, or other administrative tasks while minimising disruption to the workloads.

Practical Scenarios for Maintenance

Common scenarios requiring Kubernetes maintenance mode include:

• Firmware Upgrade on Nodes: Updating the firmware to ensure compatibility and performance.
• Cluster Upgrade: Upgrading Kubernetes to a newer version.
• OS Upgrade of Nodes: Updating the operating system for security patches and new features.
• Hardware Replacement: Replacing faulty or outdated hardware components.

For any of the above mentioned scenarios node reboot is a must.

Let’s consider a situation where you have 1 master node and 2 worker nodes. In Each worker node, there are pods running on it.

Scenario: Worker Node 1 Goes Down

When worker node 1 goes down, the pods running on it become inaccessible. Here’s what happens:

Impact on Pods:

• If a pod from a ReplicaSet is running on worker node 1, Kubernetes ensures high availability.
• If a standalone pod is running on worker node 1, users accessing this pod are impacted as the application becomes inaccessible.

ReplicaSet Behavior:

• In this scenario, there is a ReplicaSet with two pods (ReplicaSet Pod 1 and ReplicaSet Pod 2).
• If one pod goes down, the ReplicaSet ensures that the other pod remains accessible, minimizing user impact.
• However, if there is an additional standalone pod on worker node 1, users accessing this pod are highly impacted.

How Kubernetes Handles Node Failures

When a node fails, Kubernetes has mechanisms to handle the situation:

Immediate Node Recovery:

• If the node comes back online quickly (within five minutes), the kubelet process starts, and the pods come back online without significant disruption.

Prolonged Node Downtime:

• If the node remains down for more than five minutes, Kubernetes considers the pods on that node as dead.
• For pods that are part of a ReplicaSet, Kubernetes will create new pods on another available node in the cluster to maintain the desired state and ensure high availability.
• For standalone pods, they will be terminated, and it is up to the administrator to handle the recovery.

Need of Maintenance mode
A safer way to perform node reboots or gracefully shutdown a node is by deliberately draining the node of all workloads, ensuring they are moved to other nodes in the cluster. Technically, the workloads aren’t moved; rather, the pods are gracefully terminated on the current node and recreated on another node.

When you drain a node, it is also cordoned or marked as unschedulable, preventing new pods from being scheduled on it until you remove the restriction. This guarantees that the workloads are safely running on other nodes.

After draining and marking the node as unschedulable, you can reboot the node. Upon coming back online, the node remains unschedulable until you explicitly uncordon it. Note that the pods that were recreated on other nodes won’t automatically return to the original node. If any of those pods are deleted or if new pods are created, they can then be scheduled on the node.

Additionally, you can use taints and tolerations to control pod placement more precisely. By applying a taint to a node, you can repel pods that do not have a matching toleration, ensuring that only specific pods can be scheduled on that node.

Steps to perform maintenance mode on a node using kubectl

1. Cordon the node( i.e marked as unschedulable)
   prevents new pods from being scheduled on the node.

kubectl cordon <node-name>

2. Drain the workloads for the node.

kubectl drain <node-name> --ignore-daemonsets --delete-local-data

Ignore DaemonSet Pods: Ensure that the drain command does not attempt to evict DaemonSet pods, which are meant to run on specific nodes by design. Evicting DaemonSet pods could disrupt critical system services that rely on them.

Delete Local Storage: Ensure that pods using local storage are evicted even if it means losing the local data. This is important when local data is temporary or can be reconstructed, and you need to proceed with maintenance without being blocked by these pods.

Perform reboot tasks

After the node is drained and marked as unschedulable, you can reboot it. When the node comes back online, it remains unschedulable until you explicitly uncordon it. Note that the pods that were recreated on other nodes won’t automatically return to the original node. If any of those pods are deleted or new pods are created, they can then be scheduled on the node.

Uncordon the node( i.e marked as schedulable)

kubectl uncordon <node-name>

Best Practices for Cluster Maintenance

• Schedule During Low Traffic: Perform maintenance during periods of low usage to reduce impact.
• Use Pod Disruption Budgets (PDBs): Ensure PDBs are configured to balance availability and maintenance needs.
• Monitor Cluster Health: Continuously monitor the health of the cluster before, during, and after maintenance.