Customizing GKE nodes labeling

A term label usually describes a key/value pair that is attached to some object. In Kubernetes, labels are key/value pairs that are attached to manifested resources, such as pods. The Kubernetes labels can be used to organize and to select subsets of objects. Google Cloud labels (further in the post referenced as Cloud labels) are mainly used to categorize resources in billing reports in order to simplify analysis of spending costs. When the term label is used with GKE service it sometimes creates an ambiguity and confusion. The GKE documentation that referencing them as “cluster labels” may contribute to the confusion. In essence, the cluster labels are Cloud labels that are defined on a cluster level. If there is a need for usage analysis on a more gradual level, GKE provides a cluster resource usage mechanism that tracks resource usage by Kubernetes objects based on namespaces or Kubernetes labels. This mechanism is not integrated with billing reports and requires additional cost calculations.

Sometimes there is no need for cost analysis on the level of Kubernetes objects. It is enough to track the spending costs on the level of individual Kubernetes nodes. A straightforward solution would be to “export” Kubernetes labels of the cluster nodes to Cloud labels of GCE instances that run the nodes.

An algorithm

In Kubernetes the cluster nodes are labeled to store metadata about underlying hosts, such as CPU or Operating System family and to support affinity or anti-affinity deployment scenarios. Nodes also have annotations and taints that can be utilized for the same goal. For simplicity this solution will use only node’s labels.

An input of the algorithm is a list of label keys. The algorithm would read all the labels of the node and compare them vs. the list. For each matching key, it will create a Cloud label with the same key/value pair and set it to the GCE instance that runs the node.

The implementation of the algorithm is done using Python 3.8 but can be easily ported to other languages.

Read Kubernetes node labels

Node labels are read using Kubernetes API. While it is possible to use Kubernetes client library, a simple HTTP request will do:

Following the guidelines, the code uses kubernetes.default.svc hostname to call the API server. The authorization credentials of the service account associated with the pod can be read from an auto-mounted token file. The account has to have read permissions on the cluster resource “nodes”. The node name can be exposed through environment variables of the pod. The response contains JSON with all node properties. The JSON is the same as one returned by a command: kubectl get node {NODE_NAME} -o=json.

Update Cloud labels of GCE instance

Setting Cloud labels to GCE instances also can be implemented using HTTP requests. But the simplest way would be to use Google client libraries for Python. The implementation uses googleapiclient.discovery and google.authpackages. The initialization uses credentials of the Cloud service account attached to GKE node VMs.

The setLabels API has to be used to set Cloud labels to a GCE instance. Beside a set of labels the call arguments include a project id, zone name, instance name and a current label fingerprint. GKE uses the same names for cluster nodes and GCE instances that host the nodes. The zone name is stored as a Kubernetes label with the failure-domain.beta.kubernetes.io/zone key. It can be retrieved from the node labels. To get current Cloud label fingerprint an additional API call is required.

Deploying solution

The solution implementation consists of the two files: main.py and requirements.txt. It can be packaged as a container image. As an alternative, the files can be loaded into a config map and the map can be mounted to a pod’s container as a directory. The alternative approach avoids the need of building/managing the container image. The config map can be created directly from the files using the following command:

kubectl create configmap node-labeling-files --from-file=main.py \
  --from-file=requirements.txt

The deployment has to use a service account that allows it to read the “nodes” resource via Kubernetes API. It is a recommended practice to create one.

A daemonset is the best suited for running the solution since it runs an instance of the pod on each node of the cluster. The pods in the daemonset mounts the files from the configMap and defines the preconfigured Kubernetes service account. The command arguments configure the solution to “export” the Kubernetes label with the key “workload”:

Recommendations about best practices

The presented solution is for demo only. It is a good security practice to run Kubernetes workloads with a designated Cloud service account that are linked to the Kubernetes service account of the workload using Workload Identity. The use of the service account that is attached to the cluster’s GCE instances is considered less secure.

Mind that if you try this solution on a GKE cluster with enabled Workload Identity, you will not be able to use the Cloud service account attached to the GCE instances.

The daemonset and other objects should be deployed to a dedicated namespace and not to the `default` namespace like it is used in the demo.

What is not there?

• The solution does not describe a configuration of Cloud service account and a use of Workload Identity.
• The solution does not cover labeling persistent disks that are mounted to GCE instances.
• The solution does not support changing cluster node labels in real time.

Wrap up

• It is very simple to call Kubernetes API from a pod. It requires a small setup of the service account with appropriate permission and can be done without additional libraries or frameworks.
• Same goes for calling Google API although the use of Google client libraries makes code much more compact and easy to implement.
• The solution does not require management of authorization credentials.
• The full code of the solution demo can be found in github.