How to Perform Security Checks on Kubernetes Manifests in a CI/CD Pipeline

A guide to scanning Kubernetes Manifests for preventing unusual permissions before deploying them to Kubernetes Cluster

Introduction

In this article, I will introduce a method for checking Kubernetes manifests before deploying them into a cluster. Prior to deployment, you can use specialized tools to scan for potential security vulnerabilities. Integrating these tools into your CD pipeline can help reduce the risk of deploying a vulnerable pod.

Prerequisite

To begin, you’ll need to install the following tools:

• Helm
• Kubectl-slice
• Git
• Conftest

You can install them using the commands below:

#!/bin/bash

sudo apt install git wget -y

########## installing helm ##########
wget -c https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz -O helm.tar.gz
tar -xvzf helm.tar.gz
sudo mv linux-amd64/helm /usr/bin/

########## installing kubectl-slice ##########
wget -c https://github.com/patrickdappollonio/kubectl-slice/releases/download/v1.2.6/kubectl-slice_linux_x86_64.tar.gz -O kubectl-slice.tar.gz
tar -xvzf kubectl-slice.tar.gz
sudo mv kubectl-slice /usr/bin/

########## installing conftest ##########
wget -c https://github.com/open-policy-agent/conftest/releases/download/v0.42.1/conftest_0.42.1_Linux_x86_64.tar.gz -O conftest.tar.gz
tar -xvzf conftest.tar.gz
sudo mv conftest /usr/bin/

Checking the manifests

Okay, let’s say we want to check the security best practices for a sample helm chart like bitname/redis. You could use the kubectl-slice tool to slice the chart into manifest files. Next, clone a set of standard policies and use them to scan the manifests for violence. This process will help you determine if the manifests are violent or not. like this:

#!/bin/bash

helm pull oci://registry-1.docker.io/bitnamicharts/redis --version 17.11.1 --untar --untardir helm-chart

helm template test ./helm-chart/redis > all_manifests.yaml

kubectl-slice --input-file=all_manifests.yaml --output-dir=manifests

git clone https://github.com/mlkmhd/kubernetes-security-policies policy

for f in manifests/*
do 
  echo "checking $f file"
  conftest test $f
done

The output of the commands above will be as follows:

checking manifests/configmap-test-redis-configuration.yaml file

20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions
checking manifests/configmap-test-redis-health.yaml file

20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions
checking manifests/configmap-test-redis-scripts.yaml file

20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions
checking manifests/secret-test-redis.yaml file

20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions
checking manifests/serviceaccount-test-redis.yaml file

20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions
checking manifests/service-test-redis-headless.yaml file

20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions
checking manifests/service-test-redis-master.yaml file

20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions
checking manifests/service-test-redis-replicas.yaml file

20 tests, 20 passed, 0 warnings, 0 failures, 0 exceptions
checking manifests/statefulset-test-redis-master.yaml file
FAIL - manifests/statefulset-test-redis-master.yaml - main - redis in the StatefulSet test-redis-master does not have a CPU limit set
FAIL - manifests/statefulset-test-redis-master.yaml - main - redis in the StatefulSet test-redis-master does not have a memory limit set
FAIL - manifests/statefulset-test-redis-master.yaml - main - redis in the StatefulSet test-redis-master doesn't drop all capabilities
FAIL - manifests/statefulset-test-redis-master.yaml - main - redis in the StatefulSet test-redis-master has a UID of less than 10000
FAIL - manifests/statefulset-test-redis-master.yaml - main - redis in the StatefulSet test-redis-master is not using a read only root filesystem
FAIL - manifests/statefulset-test-redis-master.yaml - main - redis in the StatefulSet test-redis-master is running as root
FAIL - manifests/statefulset-test-redis-master.yaml - main - redis in the StatefulSet test-redis-master should not be configured to live in the default namespace
FAIL - manifests/statefulset-test-redis-master.yaml - main - redis in the StatefulSet test-redis-master should use imagePullPolicy=Always

20 tests, 12 passed, 0 warnings, 8 failures, 0 exceptions
checking manifests/statefulset-test-redis-replicas.yaml file
FAIL - manifests/statefulset-test-redis-replicas.yaml - main - redis in the StatefulSet test-redis-replicas does not have a CPU limit set
FAIL - manifests/statefulset-test-redis-replicas.yaml - main - redis in the StatefulSet test-redis-replicas does not have a memory limit set
FAIL - manifests/statefulset-test-redis-replicas.yaml - main - redis in the StatefulSet test-redis-replicas doesn't drop all capabilities
FAIL - manifests/statefulset-test-redis-replicas.yaml - main - redis in the StatefulSet test-redis-replicas has a UID of less than 10000
FAIL - manifests/statefulset-test-redis-replicas.yaml - main - redis in the StatefulSet test-redis-replicas is not using a read only root filesystem
FAIL - manifests/statefulset-test-redis-replicas.yaml - main - redis in the StatefulSet test-redis-replicas is running as root
FAIL - manifests/statefulset-test-redis-replicas.yaml - main - redis in the StatefulSet test-redis-replicas should not be configured to live in the default namespace
FAIL - manifests/statefulset-test-redis-replicas.yaml - main - redis in the StatefulSet test-redis-replicas should use imagePullPolicy=Always

20 tests, 12 passed, 0 warnings, 8 failures, 0 exceptions

It’s worth noting that even the official bitnami/redis may not apply some best practices for specific manifests. So, it's essential to exercise caution.

you can put this check function to your CI/CD pipeline like the following diagram:

the security test inside CI/CD pipeline

What are some security best practices?

There are 20 security best practices based on the CISA Kubernetes hardening document, which can be found in this repository via the provided link. These best practices include:

• avoiding the use of latest tag in an image address
• avoiding the use of Host IPC for any pod
• avoiding the use of Host Network for any pod
• avoiding the use of Host PID for any pod
• avoiding the use of mount docker.socket in any pod
• avoiding the use of SYS_ADMIN CAP for any pod
• avoiding the use of priviledged mode for any pod
• avoiding the use of root user inside the container
• setting memory limit usage for all pods
• setting cpu limit usage for all pods
• setting drop all CAPs from any pod
• setting read only root file system for all pods
• setting allow priviledged scalation for any pods
• setting user-id greater than 10000 in any pods
• setting livenessProbe for all pods
• setting readinessProbe for all pods
• setting a namespace for any pod
• setting pod namespace other that default namespace
• setting imagePullPolicy for all pods
• setting imagePullPolicy equal to Always

you can find all of the codes in the above document on my GitHub repository:

GitHub - mlkmhd/kubernetes-security-policies: set of policies for kubernetes manifests based on…

Feedback

If you have any feedback or suggestions for improving my code, please leave a comment on this post or send me an email at mallakimahdi@gmail.com. I would greatly appreciate your contributions to help make this article better.

If you enjoyed this post, be sure to follow me to stay updated on my latest articles.

Further Reading:

Kubernetes Monitoring with OpenTelemetry

5 Microservices Challenges and Blindspots for Developers

Microservices Monitoring: Cutting Engineering Costs and Saving Time

OpenTelemetry Tracing: Everything you need to know

Evaluating distributed tracing tools: A guide