Malicious Activity Detection in Kubernetes using Falco and OpenSearch in Oracle Cloud — Part 3

In the previous 2 articles, we looked at Falco, a cloud native security tool designed to help you with threat detection, especially on Linux and cloud native systems. We also looked at deploying it on basic compute instances in Oracle Cloud and then shipping the detected events to OCI Logging and OCI OpenSearch.

In this article, we are going to use Falco on Kubernetes, more specifically, OKE, the managed Kubernetes service in Oracle Cloud:

Set up

First, let’s s provision an OKE cluster with some worker nodes. Given the (current) extra hoops we had to jump through to get Falco Kernel modules working on Oracle Linux 8, repeating this in a Kubernetes environment is not practical.

Instead, you’ll be pleased to know that when running Falco and you choose the modern BPF driver, this process is smooth and you don’t need to mess around with software collections and gcc, no sir! You can just stick it in the worker node cloud-init script and let it install and this is the driver that we’ll use in our OKE environment.

I recommend you use the Terraform module for OKE. Create a cloud-init script in the root module:

#!/bin/bash

dnf config-manager --enable ol8_developer_EPEL

dnf config-manager --disable ol8_MySQL80 --disable ol8_MySQL80_connectors_community --disable ol8_MySQL80_tools_community

rpm --import https://falco.org/repo/falcosecurity-packages.asc

curl -s -o /etc/yum.repos.d/falcosecurity.repo https://falco.org/repo/falcosecurity-rpm.repo

dnf install -y dkms falco

systemctl enable --now falco-modern-bpf  

### adjust block volume size
/usr/libexec/oci-growfs -y

timedatectl set-timezone Australia/Sydney

# DO NOT MODIFY
curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-init.sh

## run oke provisioning script
bash -x /var/run/oke-init.sh

touch /var/log/oke.done

In your terraform.tfvars, ensure you then set the cloud-init path:

worker_cloud_init            = [
  {
    content      = "./cloudinit.sh"
    content_type = "text/x-shellscript",
  }  
]

Once the cluster is created, create an OpenSearch cluster as before.

And then login to the operator host and install Falco using the Helm chart:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update


helm install falco -n falco --create-namespace \
--set driver.kind=modern-bpf --set tty=true falcosecurity/falco \
--set falco.json_output=true \
--set falcosidekick.enabled=true \
--set falcosidekick.config.elasticsearch.hostport="https://<private_ip_of_opensearch_api>:9200" \
--set falcosidekick.config.elasticsearch.mutualtls=false \
--set falcosidekick.config.elasticsearch.checkcert=false \
--set falcosidekick.config.elasticsearch.username="changeme" \
--set falcosidekick.config.elasticsearch.password="changeme" \
--set falcosidekick.webui.enabled=true \
--set tty=true 

Check the Falco has deployed successfully:

kubectl -n falco get pods
NAME                                      READY   STATUS    RESTARTS      AGE
falco-4fdk6                               2/2     Running   0             116s
falco-4xnsh                               2/2     Running   0             116s
falco-falcosidekick-85cbc6b556-2wgf9      1/1     Running   0             116s
falco-falcosidekick-85cbc6b556-6glwn      1/1     Running   0             116s
falco-falcosidekick-ui-695dbd4f84-2m4gz   1/1     Running   2 (71s ago)   116s
falco-falcosidekick-ui-695dbd4f84-4477p   1/1     Running   2 (71s ago)   116s
falco-falcosidekick-ui-redis-0            1/1     Running   0             116s
falco-k6z69                               2/2     Running   0             116s

And the Falco pod logs:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco

Wed Oct 18 09:22:59 2023: Falco version: 0.36.1 (x86_64)
Wed Oct 18 09:22:59 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed Oct 18 09:22:59 2023: Loading rules from file /etc/falco/falco_rules.yaml
Wed Oct 18 09:22:59 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Wed Oct 18 09:22:59 2023: Starting health webserver with threadiness 4, listening on port 8765
Wed Oct 18 09:22:59 2023: Loaded event sources: syscall
Wed Oct 18 09:22:59 2023: Enabled event sources: syscall
Wed Oct 18 09:22:59 2023: Opening 'syscall' source with modern BPF probe.

Testing Falco on OKE

Let’s follow the Falco guide to simulate a malicious activity by creating a pod:

kubectl run alpine --image alpine -- sh -c "sleep infinity"

Check that the pod is ready:

kubectl get pods -w
NAME                                      READY   STATUS    RESTARTS      AGE
alpine                                    1/1     Running   0             33s

And open a shell into the container to trigger a Falco event:

kubectl exec -it alpine -- sh -c "uptime"
 09:43:42 up  3:42,  0 users,  load average: 0.10, 0.13, 0.10

When we examine the Falco output:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco | grep Notice
{"hostname":"falco-4fdk6","output":"09:43:42.504522242: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=root user_uid=0 user_loginuid=-1 process=sh proc_exepath=/bin/busybox parent=runc command=sh -c uptime terminal=34816 exe_flags=EXE_WRITABLE container_id=5d79e89095eb container_image=docker.io/library/alpine container_image_tag=latest container_name=alpine k8s_ns=falco k8s_pod_name=alpine)","priority":"Notice","rule":"Terminal shell in container","source":"syscall","tags":["T1059","container","maturity_stable","mitre_execution","shell"],"time":"2023-10-18T09:43:42.504522242Z", "output_fields": {"container.id":"5d79e89095eb","container.image.repository":"docker.io/library/alpine","container.image.tag":"latest","container.name":"alpine","evt.arg.flags":"EXE_WRITABLE","evt.time":1697622222504522242,"evt.type":"execve","k8s.ns.name":"falco","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.exepath":"/bin/busybox","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root","user.uid":0}}

we see that the event is detected.

Using the sidekick UI

You can also access the sidekick UI which we enabled earlier:

kubectl -n falco port-forward svc/falco-falcosidekick-ui 2802:2802

You can then access the sidekick UI with your browser locally:

Login with admin/admin:

Take a bit of time to poke around the UI:

Viewing in OpenSearch Dashboard

Lastly, we want to see if our Falco events made it to OCI’s OpenSearch service:

Summary

In this article, we showed how to use Falco to detect runtime threats in OKE, the managed Kubernetes service on Oracle Cloud. We then used the Falcosidekick to ship the logs directly to OCI’s OpenSearch Service. Finally, we used the Sidekick UI to analyze events in our OKE cluster.