Enhancing kubernetes security with Kubearmor
Securing containerized applications is a significant concern for organizations, especially when deploying them in production environments. Container orchestration tools like Kubernetes have enabled the seamless deployment of containers, but security remains a challenge. Kubernetes provides some security options, but it’s not enough to protect against sophisticated attacks. KubeArmor, a runtime security engine, can help address these security concerns.
What is KubeArmor
KubeArmor is a CNCF (Cloud Native Computing Foundation) sandbox project that uses extended Berkeley Packet Filter (eBPF) and Berkeley Packet Filter-Linux Security Module (BPF-LSM) to secure pods and containers in Kubernetes. It leverages Linux security modules (LSMs) for policy enforcement, which are decades-old kernel technology traditionally used for host hardening. KubeArmor simplifies the use of LSMs to enforce required policies without users having to worry about the complexities of LSMs themselves.
KubeArmor utilizes technologies like eBPF, AppArmor, or SELinux to enforce policies specified by the user. By leveraging eBPF, KubeArmor can generate detailed alerts and telemetry events that include information about the identities of the container, pod, and namespace.
See how KubeArmor looks from architecture perspective
KubeArmor features
KubeArmor offers several key features to secure containerized applications. It can restrict the behavior of containers and other workloads by providing the ability to restrict specific behavior of process executions, file accesses, networking operations, and resource utilization inside the workload. It enforces security policies at runtime using Linux Security Modules (LSMs) for each workload based on the identities (e.g., labels) of given containers or workloads. KubeArmor produces alert logs for policy violations by monitoring the operations of containers’ processes using its eBPF-based monitor. It manages internal complexities associated with LSMs and provides easy semantics for policy definitions. KubeArmor allows applying policy settings at the level of network system calls, controlling interactions among containers. Moreover, it provides a Kubernetes-native security enforcement engine that allows operators to define security policies based on Kubernetes metadata and simply apply them into Kubernetes.
Examples of KubeArmor usage
Let’s start with KubeArmor installation process, it’s very simple
curl -sfL http://get.kubearmor.io/ | sudo sh -s -- -b /usr/local/bin
karmor installYou need to wait for some time while all components are running. Meanwhile, you can install a sample application to play with.
kubectl apply -f https://raw.githubusercontent.com/kubearmor/KubeArmor/main/examples/multiubuntu/multiubuntu-deployment.yamlBefore we continue, it’s important to know that by default, KubeArmor enforces security policies into pods (no need for kubearmor-policy: enabled). kubearmor-policy: enabled defines in the annotations for the case that enableEnforcerPerPod is set. If enableEnforcerPerPod is set, KubeArmor works as the audit mode by default. It means that kubearmor-policy: audited is automatically added for all pods.
Starting with the audited mode is recommended to understand how KubeArmor works and to identify the cases where permission may be denied. Therefore, all the experiments that follow will be conducted in kubearmor-policy: audited mode.
Now we can apply the first KubeArmor policy
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
name: ksp-group-1-proc-path-block
namespace: multiubuntu
spec:
severity: 5
message: "block /bin/sleep"
selector:
matchLabels:
group: group-1
process:
matchPaths:
- path: /bin/sleep
action:
BlockThis policy uses a construct process that specifies a set of matchPaths indicating the list of binaries (in this case only /bin/sleep) and the Action is Block. And let’s try to violate this policy
POD_NAME=$(kubectl get pods -n multiubuntu -l "group=group-1,container=ubuntu-1" -o jsonpath='{.items[0].metadata.name}') && kubectl -n multiubuntu exec -it $POD_NAME -- bash
# sleep 1We are still in audited mode, so sleep execution will not be blocked, but we can see how KubeArmor sees this action in logs, just run karmor logs in another terminal
$ karmor logs
local port to be used for port forwarding kubearmor-relay-6bc49fc67d-6wjk9: 32814
Created a gRPC client (localhost:32814)
Checked the liveness of the gRPC server
Started to watch alerts
== Alert / 2023-03-19 19:53:10.986075 ==
ClusterName: default
HostName: minikube
NamespaceName: multiubuntu
PodName: ubuntu-1-deployment-565b9f669d-jzqzz
Labels: container=ubuntu-1,group=group-1
ContainerName: ubuntu-1-container
ContainerID: a7d3adc3159787a54d72bf7a4dd804365f9c27b0c40c8c1b9008b1aed2026781
ContainerImage: kubearmor/ubuntu-w-utils:0.1@sha256:b4693b003ed1fbf7f5ef2c8b9b3f96fd853c30e1b39549cf98bd772fbd99e260
Type: MatchedPolicy
PolicyName: ksp-group-1-proc-path-block
Severity: 5
Message: block /bin/sleep
Source: /bin/bash
Resource: /bin/sleep 1
Operation: Process
Action: Audit (Block)
Data: syscall=SYS_EXECVE
Enforcer: eBPF Monitor
Result: Passed
HostPID: 894623
HostPPID: 894592
PID: 144
PPID: 134
ParentProcessName: /bin/bash
ProcessName: /bin/sleepHere you can see that Action is Audit (Block), it means that such action will be blocked by KubeArmor with kubearmor-policy: enabled annotation and you’ll see Permission Denied message.
Let’s see a few more examples of KubeArmor usage to understand how powerful it is.
Previous example was about process blacklisting, now let’s create a whitelisting policy. We can isolate process and deny everything else. Like this
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
name: allow-specific-process
namespace: web
spec:
action: Allow
file:
matchDirectories:
- dir: /
recursive: true
process:
matchPaths:
- fromSource:
- path: /usr/sbin/apache2
path: /bin/sh
- path: /usr/sbin/apache2
selector:
matchLabels:
app: web
severity: 1In the next example we’ll setup alerts about unwanted actions in you pods, like this
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
name: ksp-audit-cis-nginx-2.3.4
namespace: web
spec:
tags: ["Nginx", "CIS","cis-nginx-2.3.4"]
message: "Notification! core dump directory has been accessed"
selector:
matchLabels:
app: nginx
file:
severity: 4
matchDirectories:
- dir: /var/log/nginx/
ownerOnly: true
recursive: true
action: Auditand do many and many more.
Hardening
KubeArmor offers a key advantage by providing hardening policies for frameworks such as CIS, MITRE, NIST-800–53, and STIGs out-of-the-box. This eliminates the need for extensive research and policy creation, allowing you to apply the policies directly to your workloads and benefit from enhanced security. Furthermore, KubeArmor presents these policies in the context of your workload, enabling you to evaluate how they will be implemented and the potential impact on your system. As a result, you can make informed decisions about which policies to apply and balance security needs with operational functionality.
Let’s see how it works
$ karmor recommend -n multiubuntu
INFO[0001] pulling image image="kubearmor/ubuntu-w-utils:0.1"
0.1: Pulling from kubearmor/ubuntu-w-utils
Digest: sha256:b4693b003ed1fbf7f5ef2c8b9b3f96fd853c30e1b39549cf98bd772fbd99e260
Status: Image is up to date for kubearmor/ubuntu-w-utils:0.1
INFO[0011] dumped image to tar tar=/tmp/karmor1935324745/mXfSQRJR.tar
Distribution ubuntu
INFO[0017] No runtime policy generated for multiubuntu/ubuntu-1-deployment/kubearmor/ubuntu-w-utils:0.1
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-maint-tools-access.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-trusted-cert-mod.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-system-owner-discovery.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-write-under-bin-dir.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-write-under-dev-dir.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-cronjob-cfg.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-pkg-mngr-exec.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-k8s-client-tool-exec.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-remote-file-copy.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-write-in-shm-dir.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-write-etc-dir.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-shell-history-mod.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-file-system-mounts.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-access-ctrl-permission-mod.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-system-network-env-mod.yaml ...
created policy out/multiubuntu-ubuntu-1-deployment/kubearmor-ubuntu-w-utils-0-1-file-integrity-monitoring.yaml ...
output report in out/report.txt ...
Deployment | multiubuntu/ubuntu-1-deployment
Container | kubearmor/ubuntu-w-utils:0.1
OS | linux
Arch | amd64
Distro | ubuntu
Output Directory | out/multiubuntu-ubuntu-1-deployment
policy-template version | v0.1.9
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| POLICY | SHORT DESC | SEVERITY | ACTION | TAGS |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-maint- | Restrict access to maintenance | 1 | Audit | PCI_DSS |
| tools-access.yaml | tools (apk, mii-tool, ...) | | | MITRE |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1- | Restrict access to trusted | 1 | Block | MITRE |
| trusted-cert-mod.yaml | certificated bundles in the OS | | | MITRE_T1552_unsecured_credentials |
| | image | | | |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1- | System Information Discovery | 3 | Block | MITRE |
| system-owner-discovery.yaml | - block system owner discovery | | | MITRE_T1082_system_information_discovery |
| | commands | | | |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-write- | System and Information | 5 | Block | NIST NIST_800-53_AU-2 |
| under-bin-dir.yaml | Integrity - System Monitoring | | | NIST_800-53_SI-4 MITRE |
| | make directory under /bin/ | | | MITRE_T1036_masquerading |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-write- | System and Information | 5 | Audit | NIST NIST_800-53_AU-2 |
| under-dev-dir.yaml | Integrity - System Monitoring | | | NIST_800-53_SI-4 MITRE |
| | make files under /dev/ | | | MITRE_T1036_masquerading |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1- | System and Information | 5 | Audit | NIST SI-4 |
| cronjob-cfg.yaml | Integrity - System Monitoring | | | NIST_800-53_SI-4 |
| | Detect access to cronjob files | | | CIS CIS_Linux |
| | | | | CIS_5.1_Configure_Cron |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-pkg- | System and Information | 5 | Block | NIST |
| mngr-exec.yaml | Integrity - Least | | | NIST_800-53_CM-7(4) |
| | Functionality deny execution | | | SI-4 process |
| | of package manager process in | | | NIST_800-53_SI-4 |
| | container | | | |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-k8s- | Adversaries may abuse a | 5 | Block | MITRE_T1609_container_administration_command |
| client-tool-exec.yaml | container administration | | | MITRE_TA0002_execution |
| | service to execute commands | | | MITRE_T1610_deploy_container |
| | within a container. | | | MITRE NIST_800-53 NIST_800-53_AU-2 |
| | | | | NIST_800-53_SI-4 NIST |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1- | The adversary is trying to | 5 | Block | MITRE |
| remote-file-copy.yaml | steal data. | | | MITRE_TA0008_lateral_movement |
| | | | | MITRE_TA0010_exfiltration |
| | | | | MITRE_TA0006_credential_access |
| | | | | MITRE_T1552_unsecured_credentials |
| | | | | NIST_800-53_SI-4(18) NIST |
| | | | | NIST_800-53 NIST_800-53_SC-4 |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-write- | The adversary is trying to | 5 | Block | MITRE_execution |
| in-shm-dir.yaml | write under shm folder | | | MITRE |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-write- | The adversary is trying to | 5 | Block | NIST_800-53_SI-7 NIST |
| etc-dir.yaml | avoid being detected. | | | NIST_800-53_SI-4 NIST_800-53 |
| | | | | MITRE_T1562.001_disable_or_modify_tools |
| | | | | MITRE_T1036.005_match_legitimate_name_or_location |
| | | | | MITRE_TA0003_persistence |
| | | | | MITRE MITRE_T1036_masquerading |
| | | | | MITRE_TA0005_defense_evasion |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-shell- | Adversaries may delete or | 5 | Block | NIST NIST_800-53 NIST_800-53_CM-5 |
| history-mod.yaml | modify artifacts generated | | | NIST_800-53_AU-6(8) |
| | within systems to remove | | | MITRE_T1070_indicator_removal_on_host |
| | evidence. | | | MITRE MITRE_T1036_masquerading |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-file- | Ensure successful file system | 5 | Audit | CIS CIS_Linux |
| system-mounts.yaml | mounts are collected | | | CIS_4_Logging_and_Aduditing |
| | | | | CIS_4.1.1_Data_Retention |
| | | | | CIS_4.1.14_file_system_mount |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1- | Ensure discretionary | 5 | Block | CIS CIS_Linux CIS_4_Logging_and_Aduditing |
| access-ctrl-permission-mod.yaml | access control permission | | | CIS_4.1.1_Data_Retention |
| | modification events are | | | CIS_4.1.11_system_access_control_permission |
| | collected | | | |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1- | Ensure events that modify the | 5 | Block | CIS CIS_Linux |
| system-network-env-mod.yaml | system's network environment | | | CIS_4_Logging_and_Aduditing |
| | are collected | | | CIS_4.1.1_Data_Retention |
| | | | | CIS_4.1.7_system_network_environment |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+
| kubearmor-ubuntu-w-utils-0-1-file- | File Integrity Monitoring | 1 | Block | NIST NIST_800-53_AU-2 |
| integrity-monitoring.yaml | | | | NIST_800-53_SI-4 MITRE |
| | | | | MITRE_T1036_masquerading |
| | | | | MITRE_T1565_data_manipulation |
+-------------------------------------+--------------------------------+----------+--------+---------------------------------------------------+KubeArmor applied bunch of recommended policies for all workloads in multiubuntu namespace.
Conclusion
Kubearmor is a powerful tool for securing Kubernetes-based container environments. Its flexible policy engine allows for granular control over container behavior, allowing administrators to enforce security policies tailored to their specific needs. With a wide range of pre-configured policies and support for custom policies, Kubearmor can help organizations ensure compliance with various regulatory standards and protect against a wide range of security threats. Furthermore, Kubearmor’s open-source nature makes it accessible to everyone, and its active development community ensures that it will continue to evolve and improve over time. Overall, Kubearmor is a must-have tool for any organization serious about securing their container-based workloads in Kubernetes.
