Kubernetes Runtime Security with CNCF Falco on Amazon EKS
Detection of anomaly behavior in running containers
Objective
Many companies have implemented secure CI/CD pipelines, from which the code is stored until it is deployed to production like the figure below.
Do you think that’s enough?
What if you have deployed a container image with a zero-day vulnerability like Log4Shell?
How could you detect the vulnerability in a container image before the security advisory is published?
How can you detect unauthorized access to sensitive information inside containers?
How can you detect anomaly activities inside your containers?
If you can’t find answers to the questions above, you may need to add more security layers to your cluster. So this post will talk about how to implement Kubernetes runtime security in the cluster.
Overall this post will cover,
- How to install Falco.
- How to forward the Falco event using Falcosidekick.
- How to trigger action response based on Falco events like deleting the infected pod using Fission.
What is Kubernetes Runtime Security?
Kubernetes Runtime Security is the protection of containers (or pods) against active threats once the containers are running (Sysdig).
The information security process has multiple strategies and activities, that can group into three distinct phases.
In this post, the implementation of Kubernetes runtime security will be included in the detection and response phase.
The goals of the detection are the last line of defense, negative use case, critical point when the incident happens, validation of prevention activities, and compliance evidence.
After the detection, it is important to declare the response action of the detection event. Examples of response actions are alerting, snapshotting, redeploying, and deleting.
What is Falco?
Falco is a free and open-source runtime security application developed by Sysdic, Inc. When writing this post, Falco is included in the CNCF project with incubation status.
Falco can detect and send notifications if there is activity in the container that is considered suspicious according to the rules. Falco can monitor Linux system calls using the kernel module or eBPF probe from the kernel at runtime.
Falco has several built-in rules, including:
- Privilege escalation using privileged containers.
- Namespace changes using tools like
setns. - Read/Writes to well-known directories such as
/etc,/usr/bin,/usr/sbin, etc. - Creating symlinks.
- Ownership and Mode changes.
- Unexpected network connections or socket mutations.
- Spawned processes using
execve. - Executing shell binaries such as
sh,bash,csh,zsh, etc. - Executing SSH binaries such as
ssh,scp,sftp, etc. - Mutating Linux
coreutilsexecutables. - Mutating login binaries.
- Mutating
shadowutilorpasswdexecutables such asshadowconfig,pwck,chpasswd,getpasswd,change,useradd,etc, and others.
For more details, see Falco’s documentation at https://falco.org/docs/ and a list of rules can be found at https://github.com/falcosecurity/falco/tree/master/rules.
Installation
The figure below is an overview of the architecture that will be deployed in this post.
Kubernetes Cluster
Before you can install Falco, you must first deploy a Kubernetes cluster. In this post, I choose Amazon EKS with the help of the eksctl application with CloudFormation. All deployment manifests in this post, you can clone from my public repository at https://github.com/fadhilthomas/demo-kubernetes-runtime-security
The following is the manifest used to deploy a Kubernetes cluster.
Run the following command to create a Kubernetes cluster.
fadhil@thomas:~$ eksctl create cluster -f falco-cluster.ymlAfter the cluster creation process is successful, the kube config file will be saved in the path ~/.kube/config
Fission
Fission is a serverless function framework to run active responses job inside Kubernetes Cluster. For more details, see Fission’s documentation at https://fission.io/docs/.
Run the following commands to deploy Fission on the previously created Kubernetes cluster.
fadhil@thomas:~$ kubectl create namespace fissionfadhil@thomas:~$ kubectl create -k "github.com/fission/fission/crds/v1?ref=v1.16.0"fadhil@thomas:~$ helm repo add fission-charts https://fission.github.io/fission-charts/fadhil@thomas:~$ helm repo updatefadhil@thomas:~$ helm install --version v1.16.0 --namespace fission fission fission-charts/fission-all
Check if the deployment was deployed successfully.
fadhil@thomas:~$ kubectl get pods -n fission
NAME READY STATUS
buildermgr-85cdc8bfb8-btlnn 1/1 Running
controller-59d77dd9b8-qgzx7 1/1 Running
executor-5864c7d9dc-2vgd9 1/1 Running
kubewatcher-7f9bf44c57-nvnjx 1/1 Running
mqtrigger-keda-579d76b464-6khmh 1/1 Running
router-66fdcdd546-85qds 1/1 Running
storagesvc-965b9c944-twlfk 1/1 Running
timer-6fdc8646c5-xbx92 1/1 RunningThe following is the manifest used to create ServiceAccount for Fission functions.
Run the following command to deploy the service account.
fadhil@thomas:~$ kubectl apply -f sa-falco-pod-delete.yamlFalco
There are several ways to install Falco. In this post, I use the Helm Chart. Run the following command to install Falco on the previously created Kubernetes cluster with these modified configurations below,
falco.json_output=truedriver.kind=ebpffalcosidekick.webui.enabled=truefalcosidekick.enabled=true
fadhil@thomas:~$ kubectl create namespace falcofadhil@thomas:~$ helm repo add falcosecurity https://falcosecurity.github.io/chartsfadhil@thomas:~$ helm repo updatefadhil@thomas:~$ helm upgrade --install falco falcosecurity/falco --namespace falco --set falcosidekick.config.fission.function="falco-pod-delete" --set falco.json_output=true --set driver.kind=ebpf --set falcosidekick.webui.enabled=true --set falcosidekick.enabled=truefadhil@thomas:~$ kubectl patch daemonset falco --patch '{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"falco"}],"containers":[{"name":"falco","tty":true}]}}}}' --namespace falco
Check if the deployment was deployed successfully.
fadhil@thomas:~$ kubectl get pods -n falco
NAME READY STATUS
falco-falcosidekick-df77f7458-d66pl 1/1 Running
falco-falcosidekick-df77f7458-lcnzm 1/1 Running
falco-falcosidekick-ui-7d6b97856d-7pz7g 1/1 Running
falco-falcosidekick-ui-7d6b97856d-fwjsx 1/1 Running
falco-falcosidekick-ui-redis-0 1/1 Running
falco-fsxwq 1/1 RunningDemo App
For testing, I will deploy dvwa which Falco will monitor. dvwa is an application that is vulnerable by design to several types of attacks such as command injection. The goal of the testing is to detect if someone is taking advantage of vulnerabilities such as command injection from the public.
Open the dvwa-app directory that you can find here https://github.com/fadhilthomas/demo-kubernetes-runtime-security/tree/main/dvwa-app.
Run the following command to deploy the manifest.
fadhil@thomas:~$ cd dvwa-app
fadhil@thomas:~$ kubectl apply -f dvwa-deployment.yamlCheck if the deployment was deployed successfully.
fadhil@thomas:~$ kubectl get pods -n dvwa
NAME READY STATUS
dvwa-app-54f998c8c5-l55hx 1/1 RunningFalco’s Active Response Function
I will deploy Falco’s active response with the Fission Function. The goal of the function is to delete the infected pod.
Open the falco-fission directory that you can find here https://github.com/fadhilthomas/demo-kubernetes-runtime-security/tree/main/falco-fission.
Run the following command to deploy the manifest.
fadhil@thomas:~$ cd falco-fission
fadhil@thomas:~$ fission spec applyCheck if the function was deployed successfully.
fadhil@thomas:~$ fission pkg list
NAME BUILD_STATUS
falco-pod-delete-d18f6a0b-e5a1-4275-9471-38d684ac4dfe succeededSimulation and Testing
- Detect Spawn Shell in The Pod
Run the following command to spawn the shell inside the dvwa-app pod.
fadhil@thomas:~$ kubectl exec -it dvwa-app-54f998c8c5-7b74f -- sh
# command terminated with exit code 137After a few seconds, the shell is terminated automatically. Check the deployment status, it turns out that the active response function successfully deleted the pod.
fadhil@thomas:~$ kubectl get pod
NAME READY STATUS
dvwa-app-54f998c8c5-7b74f 1/1 Terminating
dvwa-app-54f998c8c5-vzkc7 1/1 Runningfadhil@thomas:~$ kubectl logs deployment/falco-falcosidekick --namespace falco
2022/10/24 01:13:01 [INFO] : Fission - Post OK (200)
2022/10/24 01:13:01 [INFO] : Fission - Function Response : OK
2022/10/24 01:13:01 [INFO] : Fission - Call Function "falco-pod-delete" OK
You can also view Falco events by opening Falcosidekick UI.
- Detect Command Injection in The DVWA Application
Open the dvwa web and input google.com; cat /etc/passwd payload. The payload will return the value of /etc/passwdbecause there is no validation of the user input, it is a command injection vulnerability.
Check the deployment status, it turns out that the active response function successfully deleted the pod.
fadhil@thomas:~$ kubectl get pod
NAME READY STATUS
dvwa-app-54f998c8c5-l7qjw 1/1 Running
dvwa-app-54f998c8c5-m8rzr 1/1 Terminatingfadhil@thomas:~$ kubectl logs deployment/falco-falcosidekick --namespace falco
2022/10/24 01:17:30 [INFO] : Fission - Post OK (200)
2022/10/24 01:17:30 [INFO] : Fission - Function Response : OK
2022/10/24 01:17:30 [INFO] : Fission - Call Function "falco-pod-delete" OK
You can also view Falco events by opening Falcosidekick UI.
Closing
Using Falco as Kubernetes Runtime Security, you can get more visibility and detect anomaly behaviors inside your cluster. Also, it’s easy to customize Falco according to your needs, because it’s free and open source.
