Pwning Kubernetes

Written by: Yu Pengfei, Mah Chia Hui

Introduction

The explosive usage growth of Kubernetes over the last decade was a huge spectacle. However, Rome was not built in a day, and just like every other major technology stack in use today, it took several decades for containerisation technology to mature before Kubernetes presented itself as the best articulation at its current stage.

As more applications are containerised, orchestration engines such as Kubernetes have become extremely popular. As an orchestration engine, Kubernetes helps to manage, scale and deploy containerised applications. That said, there are tons of different orchestration engines (such as Docker Swarm) — so why does Kubernetes remain so popular? A quick answer lies in Kubernetes being feature-rich, packed with capabilities such as self-healing and auto-scaling. On top of that, Kubernetes is implemented as a managed service by most cloud service providers (CSP). This cements its position as the “Gold standard” for orchestration engines. Kubernetes adoption is still growing, with 96% of organisations using or evaluating Kubernetes. (https://thenewstack.io/is-kubernetes-adoption-slowing/)

Mo’ Honey Mo’ Problems

With the widespread usage of Kubernetes, it is no surprise that it is also a prized attack surface that attackers are actively trying to sniff out and exploit. With its new-ish ascendance, developers also make mistakes in its implementation, configuration or deployment.

This article is the first in our series regarding Kubernetes security. It will provide an overview of some common attacks, and basic security best practices for Kubernetes. Subsequent articles will dive deeper into more complex attacks, such as HostPath Mounting, trampoline pods/nodes, etc, and their remediations.

To understand the attacks, we must first understand the Kubernetes architecture, the high-level diagram below is a good starting point. Feel free to skip to “Attack Methodology” if you are already familiar with the Kubernetes architecture and its fundamentals.

Kubernetes Architecture Diagram

Pods

Kubernetes comprises of many moving parts, and its lowest working unit is known as a POD. A pod can host multiple different containers and storage volumes. Each pod also has its unique IP address. It is also ultra-scalable as it is based on deployment blueprints (yaml file).

Nodes

To keep things simple for everyone, we will just breakdown the type of nodes to:

● Worker Nodes

● Master Node

A set of nodes forms a cluster.

Worker Nodes:

Pods reside on these worker nodes that can either be virtual or physical machines. You can think of them as host machines for pods.

Master Node:

A Master Node is the ultimate administrator over the entire cluster. It controls and manages all worker nodes, which include the following components: Kube-API server, Kube-Controller-Manager, Kube Scheduler, etc.

Namespace

A Kubernetes namespace is an isolation mechanism that isolates resources within a cluster, generally used when different teams are working on the same cluster. In the cybersecurity context, each namespace object/resource is unique to the namespace. Cross-namespace invocation of object/resource is impossible as these objects/resources are isolated within their namespaces.

Now with all this new knowledge, you should have a fundamental understanding of the Kubernetes environment. Let’s move on to the juicier side of things, how does one attack Kubernetes?

Attack Methodology

A sophisticated attacker’s methodology can be broken down into three stages.

Stage 1: Initial foothold

The primary way to gain an initial foothold is to compromise a web application or microservice hosted on the cluster. The attacker’s objective is to gain remote code execution on the application by enumerating the application and identifying exploitable vulnerabilities (e.g. log4shell, apache struts).

Stage 2: Privilege Escalation

Upon gaining a foothold into the Kubernetes cluster, an attacker would be inside a pod, which, as explained, is simply a docker container. An attacker could sniff the container for sensitive data but is limited to what is available. . As such, the attacker will look to escalate to either a pod with more privileges or to a node.

The following are some common techniques an attacker would employ:

• Sensitive Files

Upon compromising a pod, an attacker would look for credentials such as SSH keys or passwords that would enable lateral movement to other pods that may have higher privileges. Another file an attacker would go for is the service account token, which is typically found in pods that host a long-running service. This service account token is mounted on the pod for the application to access the Kubernetes API server. An attacker can use the service account token to perform malicious operations, such as creating pods in different namespaces or even reading sensitive information from the secrets resource.

Remediation:

In addition to the standard practice of not leaving sensitive credentials unencrypted in a pod, a Kubernetes administrator should adhere to the principles of least privilege when providing privileges to the service account token.

• Reachable Applications

The attacker would enumerate the network with tools like ping or nmap and get a sense of what other applications are reachable. From there, an attacker will identify and exploit the vulnerabilities in the newly discovered applications. The aim here is to gain remote code execution on applications that exist in other namespaces.

Remediation:

To manage which pods, namespaces or even external servers can reach a particular Kubernetes resource, a Kubernetes administrator can use the Kubernetes network policy resource.

• Cloud Metadata Services

Access to the cloud metadata service (e.g. AWS instance metadata) can be a major find for an attacker. The attacker would be able to extract credentials from the metadata service, which can be used to interact with other sensitive cloud services such as AWS secrets manager. The credentials could also have further access into the cluster, such as gaining access to another pod with a different namespace.

Remediation:

The Kubernetes administrator can utilise the aforementioned Kubernetes network policy resource to prevent an attacker from reaching the cloud metadata API and other applications hosted within the cluster not required for core business functions.

• Pod breakout

An attacker can break out of a pod by exploiting misconfiguration in the Kubernetes cluster. Some common misconfigurations we have identified over numerous penetration tests are overly permissive capabilities (e.g. SYS_ADMIN), pod running in privileged mode and allowing host path mounts. An attacker can utilise the misconfigurations to break out of the pod into other pods or, worse, to the worker node on which the pod is running..

Remediation:

The Kubernetes administrator can utilise the admission controller resource, a Kubernetes resource that validates every resource being created in the Kubernetes cluster by a defined ruleset. A resource that does not comply with the defined ruleset would fail to be created, resulting in an error.

Stage 3: Getting the crown jewels

In the Kubernetes ecosystem, the crown jewels refer to the master node. An attacker who has gained full access to the master node would essentially have complete control of the Kubernetes cluster. The attacker would repeat the techniques detailed in stages 1 and 2, to laterally move from an infected pod to the underlying node and then finally to the master node. To prevent this worst-case scenario from occurring, a Kubernetes administrator must ensure that the cluster is securely configured with some of the remediations mentioned above.

Is that all?

For a more comprehensive look at your organisation’s Kubernetes security posture, we recommend that you utilise Microsoft’s Kubernetes attack matrix. The matrix comprises major tactics that are relevant to Kubernetes security. Each one of them contains several techniques attackers can use to achieve different goals.

Conclusion

As more organisations adopt Kubernetes as the de facto tool for managing container orchestration, more sensitive data will end up on Kubernetes clusters. There is an even greater need for system administrators looking to set up their cluster to get it securely configured. An attacker with full access to the master node of a Kubernetes cluster is the last thing a CISO would want to see.

Watch out for our upcoming articles, where we will share more in-depth and specific attacks that we commonly see in the field today.

Take care and catch you on the FLIPPETY-FLIP.