Improve the security of pods running on Kubernetes

Conveyor is a data platform that manages all the invisible parts of a data factory such that you can focus on what matters: delivering great data products.

At Conveyor, we take security seriously. As a small team, however, this means we need to automate our security as much as possible. We use the AWS Security hub to automatically scan for AWS best practices using AWS config. We diligently follow up on issues, resulting in a score of more than 90% on our production environment.

Apart from constantly scanning our AWS resources we also:

• Use dependabot to scan libraries
• Scan docker images for vulnerabilities
• Use Microsoft Defender for Cloud

If you want to know more about how we handle improving our AWS security I highly suggest you follow our webinar on AWS security on 7th September at 12.30 CEST. During this webinar, I will be deep diving into the topics of:

⚡ How can you start improving your AWS security today
🛠️ How to automate your security on AWS
🍡 Different security services on AWS can offer for you

While working on improving our security, I got reminded about the security context configuration of Kubernetes pods. In this article, I will show you how to tweak these settings in order to make your applications running on Kubernetes more secure. I am not a security expert, so be sure to do your own research. However, I can guarantee that the following settings will improve your security when using Kubernetes.

Security Context

In Kubernetes, most settings regarding security for pods can be set through the security context of the container. By default, Kubernetes will not restrict your pods too much. This is done to maximize compatibility and deliver a great user experience. Unfortunately, these settings are too lenient and make it easier for an intruder to exploit pods running on Kubernetes.
I will discuss 4 ways to improve the default settings.

Read-only root file system

The readOnlyRootFilesystem setting is self-explanatory as it mounts the container’s root file system as read-only.

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  containers:
  - name: sec-ctx-demo
    image: busybox:1.28
    command: [ "sh", "-c", "sleep 1h" ]
    securityContext:
      readOnlyRootFilesystem: true

Setting the readOnlyRootFilesystem flag to true reduces the attack surface of your containers since an attacker cannot manipulate the executables of your container, nor make any other changes to your root file system.

Most of the time, enabling this setting worked without issues. However, some of our applications were writing temporary files and thus needed a location to write to. We solved this by mounting an emptyDir in Kubernetes as this was a very simple change.

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  containers:
  - name: sec-ctx-demo
    image: busybox:1.28
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: tmp
      mountPath: /tmp
    securityContext:
      readOnlyRootFilesystem: true
  volumes:
  - name: tmp
    emptyDir: {}

With this minor change, we could enable the read-only root filesystem on 99% of our pods. For the remaining 1% we needed to rewrite a small part of the application. If you don’t need the /tmp mount it’s better not to add it, so always test without it first.

Run as a non-root user

Running your container as a non-root has many security advantages, and can help against a so-called container breakout attack, which is described in this article in more detail.

The following example configuration runs your container as non-root.

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
  containers:
  - name: sec-ctx-demo
    image: busybox:1.28
    command: [ "sh", "-c", "sleep 1h" ]
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000

There is a bit more to unpack here, as you can see security context can be set both on pod- and container-level. The pod level security context does not support the same settings as the container level security context. Luckily, they both support the runAsNonRoot and the runAsUser setting. In case of conflicts, the container-level security context takes precedence over the pod security context.

Only setting runAsNonRoot: true results in Kubelet are validating the image at runtime to ensure that it does not run as UID 0 (root) and it will fail to start the container if it does. So if your container runs as non-root, you don’t have to add the runAsUser setting, however, it doesn’t hurt.

We often run Golang images, which makes running as non-root rather easy. Most of our containers start from the gcr.io/distroless/base-debian10:nonroot base docker container. The executables we run inside our container can just be executed by the non-root user.

Next to our own images, we also use a lot of open-source components, which already supported running as a non-root user. For example, the spark container image provided by Conveyor is based on spark’s open-source container image and runs as non-root by default. The same is true for the official Airflow image that we use.

Run with allowPrivilegeEscalation=false

When you allow privilege escalation on a pod, nasty things can happen. This article outlines some of the attacks that can take place. In general, you do not need privilege escalation unless you are doing things like: managing the network, uploading logs files, or other system-critical things. A web application running on Kubernetes certainly does not need access to the underlying network configuration of the node. Unfortunately, this setting defaults to true. The following example shows you how to disable it:

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    allowPrivilegeEscalation: false
  containers:
  - name: sec-ctx-demo
    image: busybox:1.28
    command: [ "sh", "-c", "sleep 1h" ]
    securityContext:
      allowPrivilegeEscalation: false

You can easily change it for both the container and pod level spec and in 99% of the cases will have no impact on your application. Combining it with the previous settings will significantly reduce the attack surface on your Kubernetes cluster.

Set automountServiceAccountToken=false

This configuration is not part of the security context but can help protect against misconfiguration. By default, every namespace in Kubernetes has a service account called default, which is linked to all pods running in your namespace that have not specified one explicitly. Furthermore, the service account tokens are also mounted in these containers by default. Typically this is not a problem because the default service account has no roles attached to it. However, if someone gives it extra rights, by attaching a Kubernetes role, all pods using the default service account gain those rights. An attacker might use the default service account token to exploit this.

To prevent such an exploit, you should never attach a Kubernetes role to the default service account. Unfortunately, mistakes can always happen so it is safer to never mount the service account token if you do not plan to use it.

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  automountServiceAccountToken: false
  containers:
  - name: sec-ctx-demo
    image: busybox:1.28
    command: [ "sh", "-c", "sleep 1h" ]

Conclusion

In conclusion, we have talked about 4 easy ways in order to run Kubernetes pods more secure:

• disable automountServiceAccountToken
• disable allowPrivilegeEscalation
• run as non root
• use a readOnly root filesystem

These settings are inspired by the default rules defined in Microsoft Defender for cloud. Implementing these recommendations was easy, but hunting down all the pods that were misconfigured was more difficult. We strongly believe that security requires continuous effort and we are always looking for new ways to improve our security.

Want to know more about Conveyor?

Do you want to know more about Conveyor, take a look at our website or our YouTube channel for:

• An introduction to Conveyor
• Spark on Kubernetes in the real world

If you want to try it out, use the following link. From there you can get started for free in your own AWS account with a few easy steps.