Beginner’s Guide to Audit Kubernetes Clusters

The author is Nilesh Jayanandana Senior Software Architect at Insighture.

I have been involved with many cluster audits, mostly in private data centres, clusters spun up using Kubeadm, Kubespray, RKE etc. I thought I would share my checklist for auditing clusters and the tools used so that it would be helpful for anyone in the community.

There are 2 kinds of clusters in my opinion. Cloud managed clusters and privately managed clusters. In cloud managed clusters, you can only audit the Kubernetes resources and configs of CNI and kube-proxy. Whereas in privately managed clusters, you can check for everything.

In my audits, I would do a series of manual checks in the cluster and then use some tools to run additional checks and diagnostics. Let’s start with the manual checks.

1. Check Cluster Version
   Upgrading your cluster is really important. Make sure to keep up with the latest updates. Every version release and security patch fixes a lot of security vulnerabilities.
2. Check Secret Encryption
   Secrets in Kubernetes are basically saved in your etcd as base64 plain text unless you specifically provide an encryption config to your kube-api-server. This can be identified by looking at the flag ` — encryption-provider-config`
   More information can be found in the following link: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
3. Check Audit Logging
   Audit logging is important for your clusters so you can keep track of who did what in your cluster. It is important to enable audit logging and also ship the logs to a log aggregator as audit logs basically reside in the master nodes. Look for the following flags in kube-api-server.
   `— audit-policy-file, — audit-log-path, — audit-log-maxage, — audit-log-maxbackup, — audit-log-maxsize`
   More information can be found in the following link: https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
4. Check Kubelet API Properly Restricted
   The Kubelet API, while crucial for managing containers in Kubernetes clusters, can pose significant security risks if exposed. By checking the following flags `— anonymous-auth=false and — authorization-mode=Webhook` in your KubeletConfig, you can verify if proper restrictions are set.
5. Check Etcd Backups
   In a world surrounded by GitOps, this could sometimes be a less important check. With GitOps processes I have witnessed clusters being restored in mere seconds. But, it is still important to set backups to your etcd as a cronjob just to get through a rainy day. Some people opt to backup the entire VM itself, but I prefer to just backup the etcd simply using etcdctl. More information is available at this link: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
6. Check for Sensitive Data in Logs
   Almost all the clusters have a log aggregating system, whether it be Kibana, Loki, Stackdriver, CloudWatch or something else. Just do a quick check for sensitive data in logs by doing simple searches with keywords such as token, password, authorization etc. Kibana has a way of masking these via ingestion pipelines and I’m sure other aggregators do as well. It’s more of a developer discipline really, but human errors are bound to happen so it is better to add masking/redacting at aggregator level to be sure.

Here are a few additional checks that are non-critical:

1. Check if Data in Transit is Encrypted
   Cilium and Calico which are leading CNIs in the Kubernetes space support network encryption. There are alternate ways to encrypt data in transit for instance with the use of service meshes etc, but I prefer network encryption at a CNI level. I have encryption experience with Cilium and Calico but I believe other CNIs have this option as well.
2. Check if Runtime Security is Enabled
   Runtime security checks your containers at runtime and sees if they are doing something they are not supposed to. You can configure runtime security with seccomp, apparmor policies or go with a tool like Falco which would give you a lot of options in terms of monitoring these issues.
3. Check for Continuous Image Scanning
   We run containers in Kubernetes using various sources. These include containers that we build and third-party containers we install from Helm charts and other sources. We can of course scan the images that we build in our pipelines and vet them. However third-party containers and even the images that we build might have vulnerabilities that could be identified at a later stage when new vulnerabilities are found. Using something like a trivy-operator in your cluster to scan images continuously could help you understand and get alerted when there are vulnerabilities in your images among other things.
4. Check for Network Policies
   Check the types of workloads running the cluster and namespaces and see if there are network policies defined. It is not a must to have network policies but in a zero-trust architecture, it is recommended to run with network policies and isolate your workloads and namespaces properly.

Thus ends my manual checklist. Let’s now move on to the tools I use to grab additional data.

1. Kube-Bench — https://github.com/aquasecurity/kube-bench
   One of the best tools there is to benchmark your Kubernetes clusters and give you a comprehensive report and suggestions.
2. RBAC Tool — https://github.com/alcideio/rbac-tool
   This tool analyses all your service accounts, roles, bindings etc and gives you a visual outline of how everything is connected. Furthermore, it can provide reports on risks and generate policies from Kubernetes audit events to finetune your permissions.
3. Kubeaudit — https://github.com/Shopify/kubeaudit
   This tool enables you to scan all your Kubernetes resources and audit them for security issues.
4. Trivy Operator — https://aquasecurity.github.io/trivy-operator/v0.1.5/
   An all-in-one tool that scans your cluster end to end and provides reports on images, configurations and potential issues/risks.
5. Kube-Hunter — https://github.com/aquasecurity/kube-hunter
   Hunts for security weaknesses in the cluster externally by analysing ports, APIs etc.

By combining the manual checks and the reports from these tools, you can create a very comprehensive audit report of your Kubernetes cluster.

Hope this helps. Cheers!

👋 If you find this helpful, please click the clap 👏 button below a few times to show your support for the author 👇

Subscribe to Insights by Insighture & get similar stories in your inbox