Falco — Security at Runtime for Kubernetes

Aymen Abdelwahed
uleap
Published in
8 min readMay 9, 2022

--

When we think about cyber threats, we think about potential attacks from the outside. However, a major threat is hugely exploited by threat actors and consists of runtime manipulation; and thus internal attacks. The latter consists of planting seemingly toxic pieces of code into app servers, leading to privilege escalations and data theft/corruption.

Runtime attacks occur in the early stages of devastating attacks!

Kubernetes - Security

Kubernetes offers relatively little in the way of security tooling; it doesn’t deliver security monitoring or threat detection capabilities. It expects you — the DevSecOps guru — to observe for and respond to security-related matters yourself.

The only real resources it provides are auditing and syscalls, which allow generating logs that track resource requests to the Kubernetes API.

Kubernetes doesn’t do anything on its own to analyze the collected data (syscalls, audit logs), aid to catch potential security events, or alert on activity that may be fraudulent.

So, instead of turning to Kubernetes itself to secure the environment against threats, we’ll need to lean on external systems/tools. Those tools are runtime-based and are your way of detecting questionable activity across an entire fleet of clusters/applications.

In this post, we’ll provide a context on:

  • What is Runtime Security?
  • Why & How Falco can assist DevSecOps teams in securing their environment?
  • Architecture & Core Components of Falco: How to build a safe and resilient infrastructure against unwanted activities in application workloads?
  • Deploy: How to customize Falco deployment for seamless integration with our ecosystem.

Runtime Security

Several security barriers could be implemented, helping create a less risky environment, But still, not enough! One can do better by adding/enforcing a closer security layer. This is possible using “Runtime Security,”; A valuable mechanism to granularly scope what workloads run on a workspace, how they run, and enforce quick response to unexpected or unwanted changes.

Runtime Security: Yet another Security layer to proactively protect workloads!

Why Falco?

A “Frequent change” is a fundamental part of the cloud-native nowadays, continuously delivering/deploying software and infra stacks. Unfortunately, anomalous events may happen, affecting the security of a system and to which one must respond accurately. It’s not practical nor reasonable to tail the logs by hand in order to detect potential risks (hundreds/thousands of API requests each minute!!).

Instead, you’ll want to stream the event log data to a detection tool that can automatically monitor audit events, filter, and generate alerts when something looks awry.

Falco comes to the picture with a focus on offering continuous visibility on threats, detecting/preventing suspicious behavior, intrusions, and data theft in real-time, and ultimately enforcing appropriate actions when required.

In a nutshell, Falco is an intrusion detection tool that detects/alerts to potential risks in real-time, and thus maximizing the teams’ ability to find and fix Kubernetes security threats before they turn into major problems.

Falco ingests all syscalls/audit logs, process, and output all suspicious activity to a chosen tool, as described below:

Falco — High-Level Overview

With advanced visibility and granular control, Falco enables:

  • Filtering network traffic and taking action on established connections,
  • Detecting configuration drift and taking action upon any unexpected changes on a FileSystem,
  • Detect misbehavior such as unauthorized privilege-gains, and react to any unwanted binary execution,
  • and much more could be accomplished with Falco.

Architecture & Core Components

Falco is a behavioral activity monitoring tool built explicitly for containers.

We can identify 3 main areas for Falco:

  1. Events (introduced by Kernel, SysCalls interceptions, ..)
  2. Powerful Rules Engine where the stream of events is asserted
  3. And Alerts are triggered when a rule is violated

Components

  • Events/Inputs: Provides support with several types of input that vary from the powerful Syscalls, Kubernetes AuditLogs, and Cloud AuditLogs. All gathered through libscap/libinsp libraries.
  • Rules Engine/Processing: A flexible Rules Filter engine allows you to describe any host or container behavior or activity. (Runs as Kubernetes DaemonSets)
  • Rules Configuration: It enables full detection of suspicious activity defined by a set of rules loaded from a Rules-configuration. (Loaded from Kubernetes Secrets)
  • Alerts/Output: Reduce risk via immediate alerts through Falco-Sidekick to an AlertManager, Pager Duty, and more

Falco can easily be integrated with an enterprise-wide ecosystem through the usage of Falco-Sidekick (Kubernetes Deployment).

Deploy & Configure

For better isolation, the Falco community highly recommends running Falco outside of the container runtimes as a service in systemd; However, this could not be conceivable if running on a managed Kubernetes environment. The alternative is to get it up and running as privileged DaemonSets, to monitor all kinds of System events captured from each and every node part of the Kubernetes Cluster.

Root-level access is required, as Falco still needs to hook into the Kernel on the host

It is strongly advised to rely on Kubernetes QoS and Priority Classes to configure Falco well, so it keeps available and schedulable in the most challenging times that a Kubernetes cluster could face (degraded, short in resources, etc..)

Please refer to my previous blog-post for “K8s Cluster Protection — PriorityClasses” if you require additional details.

To start a custom deployment of Falco, we will need to download/modify the chart Values file. Before we start, we should add the Falco repo/chart:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Once done, we extract and customize the values.yaml file:

helm get values falcosecurity -o yaml > values.yaml
vim values.yaml

Below is a sample configuration that I used during deployment:

falco:
# Configure JSON as Output format
jsonOutput: true
jsonIncludeOutputProperty: true
falco.jsonIncludeTagsProperty: true
# Activates the K8s Audit Log feature for Falco
auditLog:
enabled: true
falcosidekick:
enabled: true
config:
existingSecret: "falco-sidekick-outputchannels-secrets"
webui:
enabled: enabled

Update the values.yaml file as it suits your needs, then proceed with Falco’s deployment:

helm upgrade --install falco falcosecurity/falco -n falco-system

It is recommended to use ‘helm upgrade -install’ instead of ‘helm install’ as this provides better support when managing/upgrading Falco configuration:

While waiting for the deployment to finish, we can check the logs for Falco deployment:

> kubectl logs deployment/falco-falcosidekick -n falco-system

If everything is well configured, the output should look like this:

Found 2 pods, using pod/falco-falcosidekick-5c797f56c5-8zfbg
2022/03/08 09:32:23 [INFO] : Enabled Outputs : [WebUI]
2022/03/08 09:32:23 [INFO] : Falco Sidekick is listening on :2801
2022/03/08 09:32:51 [INFO] : WebUI - Post OK (200)

Falco Rules

Falco comes with a reach rule-set particularly built to match against the main compliance frameworks, such as and not limited to PCI-DSS. (As an example, several rules are tagged with PCI offer a specific coverage to the PCI-DSS framework)

Rules are to define conditions and the message to output, and are implemented in a Rules File (Loaded during deployment and installed as a K8s Secret).

Rules are asserted at runtime; When a rule is violated on a system, Falco triggers alerts related to the violation.

Falco Alerts

Out of the box, Falco provides a very limited set of output channels: {stdout, file, Syslog, spawned program, HTTP endpoint, and a gRPC API}. While this could be enough for certain use-cases, outputs are easily extendable with Falco-Sidekick. The latter offers broad integration with tools from ChatOps, Monitoring/Alerting solutions, UI, etc.

Ensure to have proper tuning of alerts rules to avoid any alert fatigue because of False-Positives. As soon as you detect a false alarm, you need to go into and normalize alerting for it.

Time to play!! Simulate a malicious activity!

Let’s get into the fun part and try to modify a running container (in 3 easy steps).

1. Open Falco-SideKick WebUI, using port-forward, and keep watching for any thrown events. A typical port-forwarding should be enough for now (Browse to http://127.0.0.1:2802/ui )

kubectl port-forward svc/falco-falcosidekick-ui -n falco-system 2802

2. Spin up a container malicious and spawn a shell:

kubectl run -it malicious --image=falcosecurity/falco-builder \
-n falco-system -- bash

Within the running container, create a file: vi malicious.c. [And modify its content with the below code — or copy/paste it from this URL]; The code basically tries to open a FileDescriptor in RW mode, modifies its content, and closes with fclose function.

#include <stdio.h>
#include <stdlib.h>

int main() {
char sentence[1000];

// creating file pointer to work with files
FILE *fptr;

// opening file in writing mode
fptr = fopen("program.txt", "w");

// exiting program
if (fptr == NULL) {
printf("Error!");
exit(1);
}
printf("Enter a sentence:\n");
fgets(sentence, sizeof(sentence), stdin);
fprintf(fptr, "%s", sentence);
fclose(fptr);
return 0;
}

Compile the file to generate a binary called sim-hack .

Falco detects all actions from spawning a shell into a container, opening a file, modifying files, and running the script. All are recorded as threats with different priorities and triggers alerts.

Let’s do better and run the script:

gcc malicious.c -o sim-hack
./sim-hack

3. Checking back the WebUI, we can observe that our suspicious activity is well detected!! A high-priority event error has been thrown and pushed to SideKick.

15:22:02.817540337: Error File below / or /root opened for writing (user=root user_loginuid=-1 command=hack parent=bash file=/program.txt program=hack container_id=c59fb04043ff image=falcosecurity/falco-builder) k8s.ns=falco-system k8s.pod=malicious container=c59fb04043ff k8s.ns=falco-system k8s.pod=malicious container=c59fb04043ff k8s.ns=falco-system k8s.pod=malicious container=c59fb04043ff
Falco-SideKick-UI — Suspicious activity

Falco — Event Generator

We can merely run a Falco Event Generator image to initiate generating dummy events so that you can watch on Falco:

kubectl run --it falco-event-generator \
--image=falcosecurity/falco-event-generator -n falco-system

Once you’ve gotten that spun up, you should see the events being generated to SideKick-UI, like in this screenshot:

Falco Playground — Rules

https://falco.org/labs/

Falco —Self Security Report / Vulnerabilities !!

Falco itself contains several known vulnerabilities; More than 690 vulnerabilities are identified in its binaries, and we hope to see them resolved very soon.

For now, several are critical/high, shown as open issues. You must be aware of this before deploying Falco into a production cluster and either accept the risk/remediate the issues or, worst-case scenario, ignore the project.

To Conclude

In this post, we scratched the surface of possibilities; We learned how to achieve continuous Runtime-Security monitoring for container-based workloads running on Kubernetes through custom integration between Falco, Falco-SideKick, WebUI, and AWS CloudWatch/PagerDuty.

The solution could be extended further by forwarding findings to a security information and event management (SIEM) toolset such as Splunk or EFK for log correlation.

Furthermore, you can start looking into Kubernetes Response Engine to perform automated remediation activities based on the generated findings using FaaS solutions.

Thank you so much for reading me!

--

--

Aymen Abdelwahed
uleap
Editor for

Is a Cloud-Native enthusiast with 13 plus years of experience. He’s continuously immersing himself in the latest technology trends & projects.