Introduction to Kubernetes Monitoring Architecture

As you know from our earlier blogs, Kubernetes (or k8s) is an open source platform for deploying and managing containerized applications at scale. It is designed to automate such container management tasks as deployment, scaling, updating, scheduling, storage provisioning, networking, and service discovery, among others. With Kubernetes, it is simple to group multiple hosts running Linux containers and turn them into a working computer cluster controlled and managed by the “intelligent” control plane maintaining the desired state of your deployments.

Although Kubernetes dramatically simplifies the deployment of containerized applications, its multi-level architecture and multiple abstraction layers (e.g., pods, services) introduce new complexities to the daily tasks of application monitoring. There are at least two reasons why traditional approaches to monitoring don’t work well with Kubernetes:

Containers in Kubernetes are intricately entangled with Kubernetes orchestration services

To provide orchestration services, Kubernetes encapsulates application containers in the abstractions known as pods. Pods ensure a shared network and storage interfaces for containers and access to various orchestration services provided by the platform. Pods can be further abstracted into services that act as load balancers distributing traffic across the backend Pods. Pods and services are typically managed by controllers that maintain the desired state of your apps and facilitate scaling and updating them. When you think of such complex architecture (we’ve only touched the tip of the iceberg), you see that any viable monitoring solution should be able to pave its way through the maze of abstractions to actual containers that provide application metrics. At the same time, a monitoring agent should not miss the bigger picture provided by various abstraction layers and the entire cluster.

Kubernetes is a distributed and fluid environment

Kubernetes is a distributed environment where numerous applications and services are spread across multiple nodes of your cluster. From here, it follows that monitoring agents should be deployed on each node and be connected to a centralized monitoring pipeline. Second, Kubernetes is a fluid environment driven and defined by orchestration tasks. Containers are scaled up and down depending on the load and moved to other nodes all the time. Moreover, the physical infrastructure can itself be dynamically scaled horizontally with new nodes added or removed automatically. That being said, static monitoring solutions designed for standalone desktop applications are not good for Kubernetes. The platform requires monitoring tools that can dynamically capture container events and be tightly integrated with Kubernetes schedulers and controllers.

These challenges lead us to a radically new approach to monitoring of containerized applications.

Kubernetes Monitoring Approach

The approach taken by Kubernetes to monitoring directly stems from the platform’s design. According to Kubernetes documentation,

“Kubernetes is not a traditional, all-inclusive PaaS (Platform as a Service) system. Since Kubernetes operates at the container level rather than at the hardware level, it provides some generally applicable features common to PaaS offerings. … However, Kubernetes is not monolithic, and these default solutions are optional and pluggable. Kubernetes provides the building blocks for building developer platforms, but preserves user choice and flexibility where it is important.”

In other words, Kubernetes is designed to be extensible and pluggable. You can design any plugin, extension, and API compatible with the Kubernetes networking and storage standards and add them to gain visibility in Kubernetes.

At the same time, Kubernetes is not a barebones platform. When it comes to monitoring,

“It provides some integrations as proof of concept, and mechanisms to collect and export metrics.”

These mechanisms are themselves implemented as extensions and plugins running on top of Kubernetes cluster components. One of the most important of them is cAdvisor — a built-in Kubernetes monitoring tool.

cAdvisor

In order to perform various orchestration tasks (e.g., scheduling, managing container resource requests and limits, Horizontal Pod Autoscaling), cluster components such as scheduler, Kubelet, and controllers need to collect and analyze cluster metrics. Specifically, they need to know how much CPU, RAM and network resources are currently consumed by various applications and components of the cluster. Kubernetes ships with the built-in monitoring solution that provides such metrics — cAdvisor.

This tool is a daemon that collects, processes, and exports information about containers. In particular, cAdvisor keeps information about container historical resource usage, resource isolation parameters, network statistics, etc. cAdvisor is a part of the Kubelet binary, which makes it tightly coupled with Kubelet. As you remember, the Kubelet is the primary “node agent” responsible for running and managing containers on nodes in accordance with PodSpecs (see the image below).

Kubelet pulls container resource usage metrics directly from cAdvisor and makes decisions based on them. It also exposes the aggregated pod resource usage stats via a REST API.

Kubernetes users can access cAdvisor via a simple UI that typically listens on port 4194 on most Kubernetes clusters. For example, if you are using Minikube you can start cAdvisor with:

minikube start --extra-config=kubelet.CAdvisorPort=4194

To get access to the cAdvisor Web UI, you can then:

open http://$(minikube ip):4194

Native Monitoring since 1.6 and 1.8: Metrics API and Metrics Server

cAdvisor is the in-tree monitoring product provided as a part of Kubernetes binary. It was a necessary component in the earlier Kubernetes versions, but the overall tendency is to move towards detaching add-on monitoring components from upstream Kubernetes components. The extensibility mindset which Kubernetes is known for was embodied in the recent changes to Kubernetes monitoring architecture.

In particular, since 1.6 and 1.8 releases, Kubernetes moved to a new monitoring architecture based on Resource Metrics API and Custom Metrics API. These APIs are the part of the effort to standardize the way Kubernetes cluster components and third-party monitoring solutions access Kubernetes metrics generated by cAdvisor and other low-level metrics components.

A new monitoring architecture outlined in the Kubernetes proposal consists of the core metrics pipeline exposed by the Resource Metrics API and the monitoring pipeline for third-party monitoring solutions. The goal of this architecture is to provide a stable, versioned API that core Kubernetes components can use along with a set of abstractions for custom/add-on monitoring applications to easily access Kubernetes core and custom metrics. Let’s briefly discuss the design of these two pipelines.

Core Metrics Pipeline

A core metrics pipeline consists of the following components:

• Kubelet. Provides node/pod/container resource usage information (cAdvisor will be slimmed down to provide only core system metrics). Kubelet acts as a node-level and application-level metrics collector as opposed to cAdvisor responsible for cluster-wide metrics.
• Resource estimator. Runs as a DaemonSet that turns raw usage values collected from Kubelet into resource estimates ready for the use by schedulers or HPA to maintain the desired state of the cluster.
• Metrics-server. This is a mini-version of Heapster (Heapster is now deprecated) that was previously used as the main monitoring solution on top of cAdvisor for collecting Prometheus-format metrics. Metrics-server stores only the latest metrics values scraped from Kubelet and cAdvisor locally and has no sinks (i.e., does not store historical data).
• Master Metrics API. Metrics Server exposes the master metrics API via the Discovery summarizer to external clients.
• The API server. The server responsible for serving the master metrics API.

A core metrics pipeline is to be used by core system components such as scheduler, horizontal pod autoscaler, and simple UI components (e.g., kubectl top ). Although the core metrics pipeline does not include long-term metrics collection and storage, it provides a useful interface to implement it using Prometheus exposition format.

Monitoring Pipeline

The core metrics pipeline does not support third-party integrations. However, Kubernetes implements a monitoring pipeline interface specifically designed for full monitoring solutions (e.g., Prometheus). This approach removes the need to maintain Heapster as the integration endpoint for every metrics source and feature.

The key part of the monitoring pipeline architecture in Kubernetes is Custom Metrics API. With this API, clients can access both core system metrics and application metrics. This API should be implemented by monitoring pipeline vendors, on top of their metrics storage solutions.

Data collected by monitoring pipeline can contain the following metrics types:

• core system metrics
• non-core system metrics
• service metrics from application containers
• service metrics from Kubernetes infrastructure containers.

Among popular monitoring solutions that can implement Custom Metrics API one should mention Prometheus, Heapster, and various proprietary APM solutions like Datadog and Dynatrace. Full monitoring pipelines are the subject of the second part of the Monitoring series.

Deploying Metrics Server

To get an idea of how Kubernetes core metrics pipeline works, let’s try to run the add-on Metrics Server. To get the server up and running, you’ll first need to configure the aggregation layer. This layer is a new feature in Kubernetes 1.7 that allows Kubernetes apiserver to be extended with additional non-core APIs. Once the API and your add-on server are registered with the aggregation layer, the aggregator will be able to proxy relevant requests to these add-on API servers allowing you to serve custom API resources.

To configure the aggregation layer for your Metrics server, you’ll need to set a number of flags on the kube-apiserver:

--requestheader-client-ca-file=<path to aggregator CA cert>
--requestheader-allowed-names=aggregator
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
--proxy-client-cert-file=<path to aggregator proxy cert>
--proxy-client-key-file=<path to aggregator proxy key>

Notice that for the first flag, you’ll need to obtain a CA certificate if your cluster administrator has not provided you with that. You can find more information about these flags in the official kube-apiserver documentation.

After you’ve configured the kube-apiserver, the next step is to deploy the metrics-server mentioned above. This can be done with the deployment manifests from the metrics-server GitHub repository. For more information about metrics-server, see the official metrics-server repository.

You can deploy the Metrics server directly from the metrics-server-master/deploy/1.8+/ directory of the downloaded repository:

kubectl create -f metrics-server-master/deploy/1.8+/
clusterrolebinding.rbac.authorization.k8s.io “metrics-server:system:auth-delegator” created
rolebinding.rbac.authorization.k8s.io “metrics-server-auth-reader” created
apiservice.apiregistration.k8s.io “v1beta1.metrics.k8s.io” created
serviceaccount “metrics-server” created
deployment.extensions “metrics-server” created
service “metrics-server” created
clusterrole.rbac.authorization.k8s.io “system:metrics-server” created
clusterrolebinding.rbac.authorization.k8s.io “system:metrics-server” created

Once the Metrics server is deployed, we can easily access the resource metrics API with kubectl get --raw . For example, the following command returns the resource usage metrics for all nodes in your cluster.

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq
{
   "kind": "NodeMetricsList",
   "apiVersion": "metrics.k8s.io/v1beta1",
   "metadata": {
      "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
   },
   "items": [
     {
       "metadata": {
          "name": "node1.test.com",
          "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/node1.test.com",
          "creationTimestamp": "2018-08-29T13:21:06Z"
         },
       "timestamp": "2018-08-29T13:21:06Z",
       "window": "1m0s",
       "usage": {
          "cpu": "157m",
          "memory": "3845432Ki"
        }
     },
     {
        "metadata": {
           "name": "node2.test.com",
           "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/node2.test.com",
           "creationTimestamp": "2018-08-25T13:21:06Z"
         },
        "timestamp": "2018-08-23T13:21:06Z",
        "window": "1m0s",
        "usage": {
           "cpu": "431m",
           "memory": "4559560Ki"
        }
     }
   ]
}

If you are using Minikube, you can easily enable metrics-server with a few commands. First, check what add-ons are enabled:

minikube addons list
- addon-manager: enabled
- coredns: disabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- heapster: disabled
- ingress: disabled
- kube-dns: enabled
- metrics-server: disabled
- registry: disabled
- registry-creds: disabled
- storage-provisioner: enabled

As you see, metrics-server add-on is currently disabled. To enable it, run:

minikube addons enable metrics-server
metrics-server was successfully enabled

Now, you can verify that the metrics-server is running in the kube-system namespace.

Conclusion

In this article, we’ve discussed key components of the Kubernetes monitoring architecture. To sum it up, we’ve learned that the purpose of this architecture is to create an extensible and versioned metrics API for Kubernetes system components (e.g., scheduler, HPA) and custom monitoring pipelines.

The first task is accomplished by the core metrics pipeline that feeds metrics to HPA, scheduler, and other resource management systems in Kubernetes. The second task is performed by the Custom Metrics API that allows accessing both system metrics and custom metrics by third-party monitoring adapters and pipelines.

Along with the Metrics API, Kubernetes continues to ship with the built-in monitoring solutions like cAdvisor, although the overall trend is to gradually remove built-in tools from upstream components, leaving it up to third-party vendors to create monitoring solutions based on the Custom Metrics API.

In the next blogs, we’ll discuss several third-party monitoring pipelines like Prometheus and Metricbeat that can be used to provide full monitoring services for your Kubernetes clusters and applications running on them.

Originally published at supergiant.io.