Working With Taints & Tolerations In Kubernetes

What is Taints & Tolerants In Kubernetes and How Does It Work?

In general life, we often develop an affinity with people whom we like and want to engage very often, similarly, in Kubernetes, we can establish a close affinity between nodes and pods.

If you want to ensure that only a few types of pods get accommodated in any given worker Node and repelling the other pods which you don’t want to be a part of the worker Node, you can use the concept of

“Taints & Tolerations ”

Let’s Understand with mosquito & human analogy:

Suppose you don’t want the mosquito to bite you, in that case, what do you do? You make use of mosquito repellant or apply Odomos(mosquito repellant gel).

Wherever on your body, you apply Odomos that part becomes “Tainted”. Our mosquito is non-tolerant to this type of taint, so whenever they will come closer to the tainted body part they will not be able to tolerate and will fly away.

So our body here is “Tainted ” and Mosquito is “Non-tolerant ”

But if any other type of bug comes which is tolerant to the applied taint (repellant)they will be able to sit on our body.

So in this case body is “Tainted ” other bug is “Tolerant ”.

With this analogy, now we can understand that some types of Nodes of are tainted to repel some non-tolerant type of Pods, this pod will not be abe able attach to such Nodes, while some other type of Pods which are tolerant to the tainted worker Node, will be able to reside in that Node.

Taints & Tolerations In Kubernetes Context:

In the case of Kubernetes

• Node: is the person on which the bug wanted to sit. Node her is generally tainted
• Pod: is the Bug/Mosquito Itself. Pod here is generally made tolerant

Taint and tolerations are just a mechanism to impose restrictions to ensure that pods are not scheduled onto inappropriate worker nodes.

Let’s understand with the figure given below:

Explanation:

Worker NODE 1 Tainted & POD 3 Marked Tolerant:

NODE 1 here is marked as Tainted with (green color), which means that None of the pods, POD 1, POD 2, POD 3 will be able to reside on the NODE 1 until any one or all of them are marked tolerant to Node 1 taints.

So if we mark POD 3 to be tolerant against the tainted NODE 1 as shown in the figure above, and leave other pods, pod 1, pod 2 to be non-tolerant to Node 1 taints, here is how the situation may look like:

Explanation:

POD 3 is allowed to reside in NODE 1 as it has been marked tolerant to it. The rest of the pods are allowed to be accommodated in NODE 2 and NODE 3.

Note!

POD 3 even though marked tolerant to NODE 1, has all the freedom to be a part of Other Nodes: Node 2 & Node 3, but pods POD 1, POD 2 will not be allowed to be accommodated in NODE 1, until they are marked tolerant to it.

So with this intuition and understanding its time to understand

How Taints Are Applied?

You add a taint to a node using kubectl taint.

For example, adding taint using kubectl taint command will have the following syntax:

kubectl taint nodes <node name >key=value:taint-effect

Here,

• taint: is the command to apply taints in the nodes
• nodes: are set of worker nodes
• node name: is the name of the specific worker node, on which taint has to be applied, it has a key-value pair
• key-value pair: it is used to specify which application type in the pod will this node may be attached
• taint-effect: This is used to define how the pod will be treated if they are not tolerant of the taint.

Taint effect has the following types

• NoSchedule
• PreferNoSchedule
• NoExecute

So this is how the final command will look like:

kubectl taint nodes node1 app=blue: NoSchedule

here node1 is tainted with a pod with key-value pair app=blue, where pods are mapped with a taint-effect type: NoSchedule. It means no pods will be scheduled on to the given node, node1 until it has matching tolerations

Adding Tolerations To The Pod:

In the above pod definition, we specify tolerations for a pod in the Pod’s Spec section as shown in the fig above.

tolerations:
- key: "app"
  operator: "Equal"
  value: "blue"
  effect: "NoSchedule"

This tolerations definition matches, to the below command,

kubectl taint nodes node1 app=blue: NoSchedule

where we added taint to match the pod with key-value pair app=blue.

So with this toleration, now Pod tol-pod will be allowed to be scheduled with Node which is tainted to accept the value type app= blue.

How Does Taints & Toleration Work Together:

Tolerations defined in the pod definition matches the taint (mapped with worker nodes), where it looks for matching keys and taint-effect

Remember: There are two special cases:

By default operation is of type : Equal

• If in the definition there is an empty key with operator type Exists, it matches all keys, values, and effects, which means this key-value mapped with any taint, will tolerate everything.
• An empty effect matches all effects with key, Key.

Understanding Other Taint Effects:

Taint Effect: PreferNoSchedule

tolerations:
- key: "app"
  operator: "Equal"
  value: "blue"
  effect: "PreferNoSchedule"

If we change the taint-effect type to : effect: "PreferNoSchedule",

this means it is a “preference” or “soft” version of NoSchedule -- the system will try to avoid placing a pod that does not tolerate the taint on the node, but it is not required

Taint Effect: NoExecute

To understand it better let’s take a new use case where we have multiple tolerations mapped with the same key-value pair but have a different type of taint effects

Let’s add a taint a node: node 1, as shown below

- $ kubectl taint nodes node1 app=blue:NoSchedule
- $ kubectl taint nodes node1 app=blue:NoExecute
- $ kubectl taint nodes node1 app=green:NoSchedule

Now let’s create a pod definition with two tolerations, as shown below:

apiVersion: v1
kind: Pod
metadata:
  name: tol-pod
  labels:
    env: prod
spec:
  containers:
  - name: nginx-cotainer
    image: nginx
  
  tolerations:
  - key: "app"
     operator: "Equal"
     value: "blue"
     effect: "NoSchedule"
  - key: "app"
     operator: "Equal"
     value: "blue"
     effect: "NoExecute"

Explanation:

In the above-depicted situations, our pod will not be scheduled onto the node, node1 as there is no toleration matching the third taint.

But, if the pod is already running in the node, it is able to continue doing so, in case the node is newly tainted, as the third taint is the only one of the three tolerations, which is not defined to be handled by the node.

Generally, if any node is having a taint of type NoExecute and any pod that has not developed, tolerations to handle the taint , will be evicted(if they pre-exist before node was tainted ) but any pods that do tolerate the taint will never be evicted.

tolerationSeconds:

If we want to instruct pod with taint-effect: NoExecute , to stay bound to the Node which was tainted and has been forced to evict, for some particular duration we can use an optional tolerationSeconds field, in the tolerations definition, as shown below

tolerations:
- key: "app"
  operator: "Equal"
  value: "app"
  effect: "NoExecute"
  tolerationSeconds: 3600

Summary:

We understood

• What is Taints & Tolerations?
• Why Taints & Tolerations are Is Used?
• How to add taints onto the Worker Node?
• How to Add Toleration Onto The Pod?

I would like to close this piece with some bullet points which mentioned the use cases of Taints & Tolerations. As we have discussed so far Taints and tolerations are a flexible way to repel pods away from nodes or evict pods that shouldn’t be running, so it is worth seeing one of the important use cases of the same:

Dedicated Node Bound To A Particular User Groups/Set of Users

• If as a Kubernetes admin or DevOps architect you want to use any particular worker node in the k8s cluster, exclusively for a particular set or user group, we can do so by,

$ kubectl taint nodes nodename dedicated=groupName:NoSchedule

The pods with the tolerations will then be allowed to use the dedicated & tainted nodes in the cluster. In order for Pod to be scheduled onto the nodes with a dedicated label, it is required that Pod has a definition of node affinity where this label dedicated: groupName has to be specified.

We will learn more about Node Affinity in our Next article in this series or Taints & Tolerations.

2. Taint based Evictions:

We can use NoExecute taint effect wisely to enforce pod eviction based on Node issues which need to be handled to address resource constraints in the node or any other relevant issue related to worker Nodes.

What’s Next?

We will look into some more advanced use case of Taints & Tolerations like

• Node Affinity
• Taint based Pod Eviction use cases
• How to control specialized hardware-level specifications in the Nodes?
• and many more…..

If you loved this piece show me some support and affection by following me and clapping

Thanks for being there………..