Following on from an Intro to Kubernetes Hardening, this article will further explore Kubernetes security from the perspective of the attacker.
Table of Contents.
Control Plane
∘ Etcd
∘ Kubelet
∘ Kube-apiserver
∘ Static Pods
∘ Malicious Admission Controller
RBAC Abuse
∘ Stealing Tokens
∘ Powerful Verbs
∘ Lateral Movement
∘ Certificate Signing Requests
EKS
Other
Control Plane.
The overall management of a Kubernetes cluster is implemented by control plane components. These monitor and act on events sent to the cluster to, for example, update a service’s endpoints when a new selected pod is started. Given their significance in the cluster these are an enticing target for attackers.
Etcd.
All cluster state data, such as secrets, pod configs and node information, is stored within a etcd — a highly-available and distributed key-value data store. Typically, access to etcd is restricted, either entirely as in the cloud, or to only the kube-apiserver in traditional environments. This is enforced using TLS authentication with certificates generally found in /etc/kubernetes/pki.
With direct access to the database attackers have the equivalent of root permissions. Whilst access through the kube-apiserver is subject to standard access control, accessing etcd directly can bypass these and enable complete control over the cluster.
Most obviously, with access to etcd, attackers have the ability to read all Kubernetes resources, including secrets. These can be read directly from the database file. However, attackers can also use etcd to inject new objects within the cluster.
$ strings /var/lib/etcd/member/snap/db | less
# See all data in etcdResources within etcd are keyed based on the scheme /registry/<resource>/<namespace>/<name>. For efficiency, objects are serialised using Protocol Buffers — or protobuf. The standard tool etcdctl is used to interact with the database.
$ etcdctl get /registry/deployments --cacert=ca.crt --key=client.key --cert=client.crt --prefix --keys-only
/registry/deployments/kube-system/coredns
/registry/deployments/kube-system/nginx-ingress-controller
/registry/deployments/kubernetes-dashboard/dashboard-metrics-scraper
/registry/deployments/kubernetes-dashboard/kubernetes-dashboardThe tool Auger can read and decode the binary representations of Kubernetes objects within etcd and output them as YAML or JSON.
$ etcdctl get /registry/pods/default/test-pod | auger decode
apiVersion: v1
kind: Pod
metadata:
annotations: ...
creationTimestamp: 2024-01-04T18:58:23Z
...With these, attackers can inject new and tamper with existing objects in etcd. The control plane detects these changes within etcd and updates the cluster accordingly.
$ auger encode -f bad-pod.yaml | etcdctl put /registry/pods/default/bad-podThe wrapper Kubetcd builds on Auger to automate etcd injections and considers particular nuances involved with accessing etcd directly. For example, many fields handled by the logic of kube-apiserver require extra care, such as the uid of pods and container hashes.
Kubetcd implements a number of post-exploitation techniques. One example, is the injection of a pod with a name not matching the key it is stored at. Such a pod cannot be manipulated by the kube-apiserver using its name — see the below example.
$ kubetcd create pod persistence-pod -t template -p nonmatching-name
# Creates a pod called persistence-pod at /registry/pods/default/nonmatching-name based on the existing pod template
$ kubetcd get pods
NAME READY STATUS RESTARTS AGE
template 1/1 Running 0 2m
persistence-pod 1/1 Running 0 1m
$ kubetcd delete pod persistence-pod
# Pod not found as api queries for /registry/pods/default/persistence-podInconsistent pods can also be created by manipulating its namespace — in the below example the pod template defines a pod to run in the default namespace, but is injected into etcd to some nonexistent namespace. The result is a pod not listed by the apiserver when querying the default namespace and the nonexistent namespace is unindexed. The injected pod is only listed when all resources are listed with --all.
$ kubetcd create pod persistence-pod -t template -n hidden-namespace --fake-ns
# Creates a pod named persistence-pod at /registry/pods/hidden-namespace/persistence-pod
# The new pod is based of the existing pod template and so runs in the default namespace
$ kubectd get pods
NAME READY STATUS RESTARTS AGE
template 1/1 Running 0 2m
persistence-pod 1/1 Running 0 1m
$ kubetcd get namespaces
NAME STATUS AGE
default Active 10m
kube-node-lease Active 10m
kube-public Active 10m
kube-system Active 10m
$ kubetcl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
...
default persistence-pod 1/1 Running 0 2m
...Other relevant techniques include injecting privileged pods with a reverse shell (into sensitive namespaces) to gain access to an underlying node. Additionally, as discussed, direct access to etcd enables kubetcd to bypass admission controllers and Pod Security Admission.
Kubelet.
An instance of the kubelet runs on each worker node and is responsible for managing pods. The agent connects the worker to the control plane, communicating with the kube-apiserver to receive pod specifications and with the container runtime to register created containers.
To do so the kubelet exposes its own API, distinct from the kube-apiserver. Port 10250 is used by the kube-apiserver for pod manipulation and port 10255 exposes a read-only access. By default, the kubelet enables anonymous authentication and an AlwaysAllow authorization mode.
Whilst its endpoints are undocumented, the kubelet code is open-source. As such, its behaviour has been reverse engineered and is a common target for attackers to gain control over a node and its containers. Consider the following snippet from the kubelet source revealing some of its endpoints and parameters.
func (s *Server) InstallDebuggingDisabledHandlers() {
h := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "Debug endpoints are disabled.", http.StatusMethodNotAllowed)
})
s.addMetricsBucketMatcher("run")
s.addMetricsBucketMatcher("exec")
s.addMetricsBucketMatcher("attach")
s.addMetricsBucketMatcher("portForward")
s.addMetricsBucketMatcher("containerLogs")
s.addMetricsBucketMatcher("runningpods")
s.addMetricsBucketMatcher("pprof")
s.addMetricsBucketMatcher("logs")
paths := []string{
"/run/", "/exec/", "/attach/", "/portForward/", "/containerLogs/",
"/runningpods/", pprofBasePath, logsPath}
for _, p := range paths {
s.restfulCont.Handle(p, h)
}
}Attackers can call these endpoints to list information on running pods, metrics, kubelet configuration, etc. Requests to the Kubelet bypass the normal admission control and audit logging implemented by the kube-apiserver.
One of the most interesting endpoint to attackers is /run which exposes the ability execute commands within pods. Most obviously, this is used to inject reverse shells or read service account tokens to then use to query the kube-apiserver.
$ curl -k "https://<kubelet>:10250/runningpods"
# Lists information about running pods
$ curl -k "https://<node_ip>:10250/configz"
# Returns kubelet configuration info
$ curl -k -XPOST "https://<kubelet>:10250/run/<namespace>/<pod>/<container>" -d "cmd=<command>"
# Executes a given command within a podThe tool Kubeletctl implements a client to query the kubelet’s API and related functionality to, for example, scan for service account tokens or run commands on all containers at once.
$ kubeletctl scan token
$ kubeletctl run "uname -a; whoami" --all-podsTo mitigate the above, the kubelet should have anonymous requests disabled and implement proper authentication and authorization. This can be enabled in its configuration file. In these cases an attacker additionally requires a valid RBAC token and PKI certs.
Kube-apiserver.
The kube-apiserver exposes access to the cluster, implementing functionality to manage Kubernetes resources. Typically, communication with the API requires TLS enabled through the files passed using the arguments--tls-cert-file and --tls-private-keys.
Kubernetes supports RBAC, with credentials to access the kube-apiserver found in a variety of places such as ~/.kube, environment variable KUBECONFIG and mounted into pods at /run/secrets/,/var/run/secrets/ or/secrets. The API server can also be configured to allow anonymous access.
With access to the kube-apiserver an attacker can enumerate existing resources and, depending on the authorization of their obtained credentials, manage cluster objects. We’ll consider this further in a bit.
Static Pods.
Whilst most pods are watched and managed by the API server, instead the kubelet is responsible for managing static pods on each node. To do so it monitors a local directory which stores the configuration of static pods. This is specified by its configuration parameter staticPodPath or argument — pod-manifest-path and is often set to/etc/kubernetes/manifests.
The kubelet registers static pods with the api as mirror pods. Querying the kube-apiserver, these appear as normal pods but cannot be edited or deleted. Their names are prefixed by the node they reside on.
$ kubectl get pods -o wide
NAME .... NODE ....
<static_pod_name>-<node_hostname> <node_hostname>Having compromised a node, creating a static pods can be a stealthy way to obtain persistence. Unfortunately (for attackers), since these are managed by the kubelet their spec cannot refer to ther API objects which limits their utility.
Malicious Admission Controller.
Admission controllers are used to control resource creation within a cluster. These intercept resource requests prior to authentication and authorization, but before being persisted. Controllers may be validating, where requests are accepted or denied, or mutating, where requests are modified.
Various built-in admission controllers can be enabled through kube-apiserver's--enable-admission-plugins argument, but most relevant to attackers is the MutatingAdmissionWebhook plugin — enabled by default in 1.29. Attackers can register a new mutating admission controllers by creating a MutatingWebhookConfiguration.
With the ability to inject a new admission controller, an attacker can have requests sent to their mutating webhook server to potentially escalate privileges or establish persistence. For example, new pods could be modified to use vulnerable images or contain back-doored sidecar containers. For more on this see here and here.
RBAC Abuse.
With access to valid Kubernetes credentials and the kube-apiserver, attackers are looking to escalate their privileges. Depending on attack’s permissions these process can look very different. In this section we’ll consider several techniques for this. For more information see here, here and here.
Stealing Tokens.
Creation. Some of the most common techniques involve stealing service account tokens mounted into pods. Most obviously, with permission to create pods, an attacker can automount the token of a new service account within the same namespace and read it for themselves. Note, this does not necessarily require the ability to execute commands within a pod as an attacker can modify the commands of the pod on creation.
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: <namespace-with-service-account>
spec:
containers:
- name: nginx
image: nginx
command: ["/bin/sh"]
args: ["-c", 'sleep 360']
serviceAccountName: <service-account-name>
automountServiceAccountToken: trueExecute, Patch & Ephemeral Containers. Even without permission to create pods, attackers may be able to steal credentials similarly from existing pods with the ability to execute commands. Additionally, the ability to patch existing resources and create ephemeral containers can enable similar techniques.
$ kubectl -n <namespace> exec pod -- "cat /var/run/secrets/kubernets/serviceaccount/token"Scheduling. Various permissions can enable an attacker to influence pod scheduling (the allocation of pods to nodes). This includes the deletion of pods, eviction of pods, modification of pod statuses, updating nodes to be unschedulable and deletion of nodes. Having compromised a node, it could be possible for an attacker to steal a pod to schedule it on the controlled host and read its mounted tokens.
Secrets. Kubernetes secrets contain lots of sensitive configuration information that may be interesting to attackers, such as passwords. This can also include service account tokens, which prior to 1.24 had an associated non-expiring secret created by default. However, tokens can also be created manually.
Powerful Verbs.
Impersonate. Kubernetes supports impersonation, enabling a user to make API requests on behalf as a another user. This is implemented using additional HTTP headers and granted to users through the impersonate verb. Consider the following request making a request impersonating a service account to retrieve cluster secrets.
curl -k -v -XGET -H "Authorization: Bearer <impersonator token>" \
-H "Impersonate-User: system:serviceaccount:<namespace>:<service-account-name>" \
-H "Accept: application/json" \
https://<master_ip>:<port>/api/v1/namespaces/kube-system/secrets/With uncontrolled impersonation a user may be able to act as a user with greater privileges than they currently have.
Escalate. To limit privilege escalation, the RBAC API prevents a user from updating or creating a role with greater permissions, unless explicit permission has been granted through the escalate verb. With access to this an attacker can modify roles or cluster roles to which they have access to and grant themselves greater privileges.
Bind. As outlined in the docs, similar to escalate, the RBAC API only allows a user to update or create a role binding if they already have all the permissions in the referenced role or explicit permission has been granted through the bind verb on the particular role. With this an attacker can bind themselves to a new role with greater permissions.
Create Token. Whilst the creation of some resources, such as pods, can enable implicit escalation, permission to the verb create over the resource serviceaccounts/token enables a user to create new tokens for a given service account (with greater privileges).
Lateral Movement.
With particular privileges attackers may be able to escalate privileges outside the RBAC environment and into other nodes or pods.
Resource Creation on Nodes. Most obviously with the ability to create resources, an attacker could schedule a new vulnerable container on a particular host and exploit it to escape and gain access to the node. Note that this does not require explicit permission to create a pod. Instead a number of alternate techniques can be used to launch new containers, such as the creation of ReplicationControllers, Jobs, StatefulSets, etc.
Depending on the cluster’s admission policy and the attacker’s permissions, the access gained can vary from just read access to the file system to complete control over the node. See this repo for example manifests which cover all possible ways to create a pod and different resulting privileges.
Port Forwarding. To debug applications Kubernetes supports the ability to forward a local port to another port within a pod. This could enable an attacker to obtain access to potentially interesting applications within a cluster such as databses or vulnerable applications.
# Listen on port 8888 locally, forwarding to 5000 in the pod
$ kubectl port-forward pod/mypod 8888:5000
# Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in a pod selected by the deployment
$ kubectl port-forward deployment/mydeployment 5000 6000
# Listen on port 8443 locally, forwarding to the targetPort of the service's port named "https" in a pod selected by the service
$ kubectl port-forward service/myservice 8443:httpsEphemeral Containers. As discussed ephemeral containers can be used to steal tokens, but could be useful to gain code execution within a particular pod and access to its resources.
Nodes/Proxy. Access to the proxy subresource of nodes grants a user access to the Kubelet API. As previously discussed this can be used to gain code execution within pods running on a node and bypass access control and logging. Depending on the configuration of the Kubelet, leveraging this privilege may additionally require access to necessary PKI certs to authenticate.
$ curl -k -H "Authorization: Bearer $(token)" "https://<kubelet>:10250/pods"
# Lists information about running podsCertificate Signing Requests.
Kubernetes security relies on PKI authentication to be implemented between many of its components, such as the Kube-apiserver and Kubelet. To support this, the Kubernetes API exposes an interface to request new certificates to be signed without having to share the CA keys.
Access to the API is controlled by permissions to the CertificateSigningRequest (CSR) resource. Depending on the request it may or may not be approved. However, with the ability to create signing requests, and have them signed, attackers can escalate their privileges a number of ways.
Most obviously, an attacker can identify a user with enticing privileges and make a request for a new certificate that can then be used to authenticate as the user. However, PKI also plays a key role in adding new nodes to clusters and the communication between key control plane components. Attackers may be able to exploit these processes.
EKS.
Whilst many techniques described so far assume a traditional on-site cluster, in reality most clusters are hosted by cloud providers. In this section we’ll consider post-exploitation within EKS clusters (AWS). For GCP’s GKE see here and here.
Kueb2IAM & Kiam. In the past third-party solutions such as kube2iam and kiam were the go-to for integrating AWS IAM with Kubernetes. These implement a privileged DaemonSet to intercept calls made to the EC2 metadata service and return temporary credentials to containers.
To do so, namespaces and pods are annotated to control which roles can be assumed by a pod. An attacker can enumerate these annotations and obtain credentials from the metadata service.
$ kubectl describe {pods,namespaces} | grep iam.amazonaws.com
# Check for kiam / kube2iam annotations
$ TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
# Obtain session token to access metadata service
$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/<role>
# Obtain temporary credentials for a given roleEven without IAM having been implemented with the cluster workloads, it’s likely the node has its own instance profile which could be obtained from the metadata service by an attacker having escaped a pod.
IAM Roles for Service Accounts. The official solution to integrate IAM with Kubernetes uses OpenID Connect (OIDC). In this, Kubernetes service accounts are associated with an IAM role indicated by the eks.amazonaws.com/role-arn annotation. The service account tokens are mounted into applicable pods using volume projection and the EKS Pod Identity Webhook updates the AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables within the pod.
To assume a role the pod makes a request to AWS STS with the service account token (a JWT), which is then validated using the cluster’s associated OIDC identity provider, specified in the role’s trust policy. See here for a more visual explanation of this process.
Having compromised a pod an attacker can check the environment variables and mounts at /var/run/secrets/eks.amazonaws.com/service-account/token to assume the role associated with the service account of the pod.
AWS-auth ConfigMap. AWS uses a token authentication webhook to enable access control to EKS clusters based on IAM. One option to manage EKS cluster authentication relies on the aws-auth ConfigMap. With access to this an attacker can establish persistence by enabling access to the cluster from their own AWS account.
Other.
Whilst this article has aimed to give an overview of the Kubernetes attack surface, a number of other key areas are missing. This includes network attacks, see here and here, container escapes, vulnerable control plane versions and supply chain attacks. We’ll consider these further next time.
