Deploy your IBM Cloud Paks on specific nodes in your cluster

As you deploy your applications to a Kubernetes cluster, you might often need to ensure that workloads are confined to certain nodes — or group of nodes — so that you can control the allocation of physical resources and reduce interference with other workloads.

IBM Cloud Paks are no exception: our customers often need to limit the installation of their Cloud Paks to a specific set of nodes.

Kubernetes and the Openshift Container Platform make available a rich — and fairly complex — set of options to achieve this type of allocation. Let’s review these options quickly, and then discuss, with an example, how to best apply them to IBM Cloud Paks.

The Kubernetes pod node selector

The pod node selector is perhaps the most basic option that allows you to control where a pod will be scheduled. It works by creating a label on the target nodes, and by specifying the nodeSelector keyword in your pod specifications. For example, consider a pod with this node selector:

This pod would only be scheduled on nodes with matching labels (region: east and type:user-node).

The pod node selector provides a “hard enforcement” type of control — if no matching nodes are present, the pod won’t be scheduled. Consequently, you should add matching labels to one or more of your nodes to ensure that those pods will be scheduled.

Here are some of the considerations that apply:

• This mechanism is very simple, but it requires you to control the spec of the pods, which makes it unsuitable, in general, to control where Cloud Paks get installed.
• This mechanism is rigid — and it is getting replaced by the node affinity options (which we’ll discuss in a moment). Node affinity specifications offer a more expressive and flexible syntax.

The default cluster-wide node selector

Let’s discuss the default cluster-wide node selector. This selector is defined on the Scheduler object (normally, there’s just one instance of it in your cluster). When you specify the cluster-wide node selector, all pods, by default, will be automatically provided with the node selector as they are deployed to your cluster, unless you specified an overriding selector on the spec (either explicitly, as described earlier, or through a project-wide node selector, which we’ll discuss in the next section).
Here’s an example (the default Scheduler object is normally called cluster). Notice the defaultNodeSelector definition:

If your scheduler has that default cluster-wide node selector, any pod that doesn’t have an overriding selector will only be schedulable on nodes that are labeled accordingly.

The project-wide node selector

In a similar way, you can annotate an Openshift project with a project-wide default node selector, as shown below (you may need to add the annotation to the namespace to add it to the Project object).

In the example above, all the pods created in that cp4ba namespace will be automatically provided with a nodeSelector spec that has the type=cloud-paks value. In turn, any pod created inside cp4ba will only be schedulable on nodes that have the type: cloud-paks label (unless, of course, the pod spec has an explicit nodeSelector of its own).

You can probably start appreciating how the project-wide node selector can be a powerful and yet simple way to isolate Cloud Paks to a set of nodes, as we’ll show in the walk-through later on.

Using Node Affinity

Node affinity is similar to the pod node selector, but it provides additional flexibility. It can be used for “hard” (mandated) or “soft” (preferred) enforcement, and it has a syntax that is much richer than that of pod node selector. You still express “matching rules” against the node labels, but those rules can include complex operators.

You can have affinity rules and node selectors on the same pod. In that case, both need to be fulfilled for the pod to be schedulable.

One consideration is that affinity is again defined in the spec section of the pod: some Cloud Paks, however, provide custom resources that allow you to specify the affinity definition. We will see an example in the walk-through.

Taints and Tolerations

The last Kubernetes feature that we are going to discuss involves taints and tolerations, which, conceptually, are the “opposite” of affinity. When you “taint” a node, it will reject any pods that do not explicitly have a “toleration” for the taint.

This is a way to ensure that a certain set of nodes will never accept pods that aren’t explicitly tolerant of the taints — but, again, you need full control of the spec section of your pods to use tolerations. Therefore, you could use taints in conjunction with Cloud Paks, but only to ensure that your Cloud Paks pods won’t end up on certain nodes.

For example, you could taint some nodes this way:

Only pods that present a matching toleration will be schedulable on node worker0. Here is an example:

Clearly, if you want to use this technique, you must take care of adding tolerations to all the workloads that you want to direct to the tainted nodes. Pods generated by Cloud Paks will not present the toleration and will not be scheduled on the tainted nodes.

In the walk-through, we show a different and less invasive approach to achieve the same objective.

Walking through a concrete example:
Cloud Pak for Business Automation and Cloud Pak for Integration

Here we provide a quick walk-through of a complete example, where we install Cloud Pak for Business Automation and “corral” it on 3 specific nodes (out of a total of 5). Then we install IBM MQ from Cloud Pak for Integration and create a Queue Manager that we want to run on a specific node, using node affinity. The principles we followed are documented here.

1 — We started with a 5-nodes cluster, as illustrated below. The worker nodes are highlighted. Notice the number of pods on each. The intent is to isolate our Cloud Paks on the last three workers (worker 2, 3, and 4). Please try to keep an eye on the number of pods throughout the walk-through.

2 — We add the label use: cloud-paks to three of the nodes — those where we want the Cloud Pak to go (we picked the last 3 — worker2, 3, and 4):

3 — And we verify that the label is in place:

4 — We then create a project for the Cloud Paks — called cloud-paks by running oc new-project cloud-paks, and we add the project-wide node selector by annotating the namespace:

5 — We verify that the project gets annotated:

6 — We then create the ibm-common-services project and annotate that too with the same node selector. This step is very important: if you omit it, this project will be created automatically during the installation of the Cloud Pak, and it will be without the node selector. All the common services pods would end up scattered around the cluster, potentially on nodes where you do not want them to run.

7 — We then proceed to installing the IBM Cloud Pak for Business Automation into the cloud-paks project. We picked the “demo” deployment of Cloud Pak for Business Automation 21.0.2.5. At the end, notice that all the new pods have been placed on the three nodes we labeled:

8 — You may notice that only one pod ended up outside the pool of worker pods we allocated to the Cloud Pak (worker1 has one extra pod at this time). That’s because one of the operators was installed in an Openshift project. But all the remaining pods are either in ibm-common-services or cloud-paks, and ended up on the last three worker nodes, as desired.

9 — Now, we install the Operator for IBM MQ, from Cloud Pak for Integration using the Operator Hub. We install the operator in the same cloud-paks project, as shown in the screenshots below.

And:

10 — Notice that this operator pod also goes on one of the three worker nodes we labeled earlier.

11 — Now we create an MQ Queue Manager and we want to make sure that it ends up on a specific node within that pool of nodes. Let’s pick worker2. We annotate that node with another label (mq=allowed).

12 — We deploy the QueueManager resource with a special affinity definition, as shown below:

13 — This definition says that this Queue Manager can only run on a node that has a label called mq, no matter what its value is:

• We use the requiredDuringSchedulingIgnoredDuringExecution affinity keyword, which enables the “hard” enforcement of the affinity rule. You could use preferredDuringSchedulingIgnoredDuringExecution as well, if you don’t want the enforcement to be mandatory.
• We use the operator Exists, which indicates that any node with that label, irrespectively of its value, will be eligible. Other operators include In, NotIn, DoesNotExist, Lt, Gt.
• In addition, the pod will be added the node selector automatically, by virtue of the project-wide node selector. Therefore, this Queue Manager can only run on a node that has both the use=cloud-paks label, and the mq label.

14 — We verify that the Queue Manager comes up and ends up just where we wanted — on worker2:

15 — So far, everything we did guarantees that all the Cloud Paks you install in the namespace that we labeled end up running on those three nodes. But there is no guarantee that other pods (not related to the Cloud Paks) won’t end up on those nodes.
To ensure that those nodes will be left alone, you can use the default cluster-wide node selector. To set it up, you need to patch the Scheduler resource as follows:

16 — This way, all pods created outside the cloud-paks namespace will be given that node selector, and will be schedulable only on nodes that have that label. Therefore, we label the remaining two nodes with that label:

17 — Now, for example, we create this deployment in the default namespace:

18 — The corresponding pod ends up in one of the two nodes that we labeled with use=general:

This concludes the walk-through!

One word of caution: you might be tempted to add the cluster-wide node selector before installing the Cloud Paks. This is possible, but keep in mind that some storage providers may attempt to schedule pods as you install the Cloud Paks, and — because of the cluster-wide node selector — those pods may fail to schedule on the nodes that are reserved to the Cloud Paks. The same may happen if you add nodes to your cluster later on. In those situations, you may need to perform extra configuration steps to allow those pods to be scheduled where they are needed.

Hopefully, you obtained a good understanding of the options that are available to you to confine your IBM Cloud Paks onto a specific set of nodes, so that you can better control resource allocation and workload isolation.

Some useful references:

Kubernetes docs about assigning pods to nodes
Openshift docs about node affinity
Openshift docs about node selectors
Cloud Paks docs about pod placement