LFX Mentorship-Kubernetes: PolicyReport output for Falcosidekick

On a sweet summer afternoon in India, I got a mail of acceptance from LFX stating that I had been selected for the LFX mentorship. Fast forward to today, it did turn out to be the watershed moment in my life I expected it to be.

In the most beautiful and thrilling journey that the last three months have been; I went from having very little technical knowledge about Golang, Kubernetes, and open source contributions to absolutely falling in love with them. All of this was possible for me because of LFX Mentorship. It gave me the much-required upliftment and mentoring to begin my open source journey
The platform not only taught me how to write quality code and implement critical thinking in practice but also taught me how to communicate at work, handle commitments, plan and execute real-life projects, and most importantly for me, it taught me how to deal with presentation anxiety. Going through these 3 months of mentorship has given me a sense of belonging to the Kubernetes community ❤.

All About LFX Mentorship

If you are a beginner to open source contributions and are trying to apply for LFX Mentorship (Spring, Summer, Fall), I would like to give you some tips and resources that helped me and I believe will help you!

1. Kubernetes is vast and finding the right place to start can be overwhelming; so here is a tweet by Dims that helped me and many more find their starting point.
2. You can never go wrong with the documentation. I didn't understand the documentation in my first read and I don't think many people do either, but slowly as you learn and read more, you connect dots and understand and that is why I believe that it is very important to go through the documentation
3. Recently learned and understood, one of the best ways to start is by taking a good first issue and exploring the project
4. I was new to Golang as well so going through the documentation, tour of go and gophercises helped me a lot.
5. Now if you are planning to apply for LFX Mentorship (if you aren't; I would highly recommend you check it out), the platform gives you three sessions in a year; spring, summer, fall; to apply to a maximum of three projects per session (You can watch out for new projects here for projects from the CNCF community)
6. The first step would be to go through all the projects thoroughly and understand which projects you should apply for.
7. Certain things that I did and that I would recommend are; checking out topics/technologies that you think you might like; meeting at least 50% of the requirements; join the slack channel for the projects you are interested in; communicate on the channel to better your understanding of the project; attend meetings if the community has any line up and introduce yourself and interact :)
8. Now that you have your projects you will have to do a set of predefined tasks. This part included a resume and a cover letter for me. You must let your resume and cover letter reflect why you deserve the mentorship and how you meet the requirements). Apart from these two, I tried submitting a proposal (included with timelines and expected solutions ) with my cover letter.

Now wait and let the magic happen!

In my project, I had to develop an adapter to run Falco in any Kubernetes cluster and periodically generate/update a Policy Report custom resource.
To elaborate: I was supposed to work on a Falco adapter i.e. a program that would accept Falco outputs and give policy reports in accordance with PolicyReport CRD.

I worked on this project with my mentor Jim Bugwadia, who worked and helped me at all times and I had the constant support of the community. I could always put up doubts and errors that I faced and have members of the community help me through it. I wouldn't have been able to work and deliver the project in time without the support and guidance of my mentor and the community. Special thanks to Gus Parvin, Thomas Labarussias, Dan Pop Papandrea, and Mritunjay Sharma. Shout out to Adeniyi Stephen for being
a great work partner, check out his project Trivy adapter here.

Some Basic Understanding

Before we begin understanding our project, we will briefly discuss Kubernetes, CRD, Falco, and Falcosidekick as they will be key components of our future discussion!

Kubernetes — Kubernetes is an open source container orchestration engine for automating deployment, scaling, and management of containerized applications. The open source project is hosted by the Cloud Native Computing Foundation.

Kubernetes CRD — The CustomResourceDefinition API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify. The Kubernetes API serves and handles the storage of your custom resource.

Falco — The Falco Project is an open source runtime security tool that uses system calls to secure and monitor a system, by:

• Parsing the Linux system calls from the kernel at runtime
• Asserting the stream against a powerful rules engine
• Alerting when a rule is violated

A rule is a condition under which an alert should be generated. It is accompanied by a descriptive output string that is sent with the alert.
If I were to explain this: Falco is a tool that you would use when you want to get notified about when and what is going wrong in your system/cluster. The rules are pre-defined but also can be customized to what you want Falco to detect.

Now that we know what is Falco, let’s discuss another tool that works on top of it and makes it easier for us to get Falco outputs: Falcosidekick

Falcosidekick — A simple daemon for connecting Falco to your ecosystem. It takes Falco's events and forward them to different outputs in a fan-out way.
It works as a single endpoint for as many Falco instances as you want .
In layman’s terms: Falcosidekick makes it easier for you to get Falco outputs by providing many formats in which you can receive Falco alerts. For example, you can enable slack output to get Falco alerts as slack messages, or you can enable web UI to see graphically what alerts are being generated. Isn’t that cool!

WHY this project!?

Every policy engine provides some output, to unify the outputs by multiple policy engines and to help cluster-admin study and investigate these outputs while managing them as K8s resources using K8s management tool (kubectl, dashboard, Octant, etc.) the Kubernetes Policy WG has defined a Policy Report CRD. We use the CRD and develop the adapter for the same purpose!

How I went about my project

“Time flies” never made more sense than today. The last three months felt like an eternity then but now feels like a few minutes. In this section, I will try to cover the milestones planned and achieved in this time period.

A majority of the first month went into researching and setting the best direction to take the project in. We put quality time into our research; having a proper roadmap to the solution, and having a well-defined UX ready was a priority. This required a lot of trial and error, communicating with members in the community, going through similar work, and prior research by the community members.

Phase 1

Falco architecture to get a better understanding

We started the project by trying to figure out the kind of outputs Falco provides, and how we could get these Falco outputs to our adapter. The two options we were considering were gRPC output and HTTP output via Falcosidekick. After the required research we figured out that contributing to Falcosidekick would be the best and also the easiest way to get Falco outputs. This meant that the policy adapter would be another output for Falcosidekick. We studied the different available outputs (kubeless,OpenFaas, etc.) to understand how we would go about building ours.

Further, our research included understanding the format of Falco output provided by Falcosidekick, Falco rules, and getting an idea of mapping by studying Gus Parvin’s work in creating a sample policy report and finalizing the mapping document.

Let’s take a minute and understand what we wanted to accomplish while finalizing mapping: The PolicyReport CRD defines certain fields that need to be assigned values from the output (in our case Falco alerts) provided by the policy engine (Falco). Here’s how the mapping part of the Falco adapter came to look like a month after the document was finalized.

code snippet showing Result that is returned after mapping

Finally, we prepared a document to jot down the points about user experience. We were now ready to start coding.

Phase 2

We started working on a new output for Falcosidekick locally after taking inspiration from the other outputs. This is how the code initially looked.

code snippet showing how we began :)

Thomas helped us set up and gave us a fake alert generator for Falco that he had created, this proved to be very useful in testing as it helped generate fake alerts of any kind that we needed as quickly as we needed. Now we set up our Kubernetes client that would help us interact with PolicyReport CRD and create/update policy reports.
We needed to interact with generated code to create/update reports, we imported this from kubernetes/wg-policy-prototypes (that Mritunjay had earlier worked on) instead of generating it in Falcosidekick, and with this, we created our first report, with basic mapping and the ability to just create and not update. Following multiple attempts and inspiration from Kube-bench-adapter, we were now able to update events in one report i.e. a cluster-wide report.

code snippet showing initial cluster report create and update logic

Phase 3

At this point, we had one report being created and updated, but we couldn’t have just one big report(there is a limit to how big one report can be) or too many reports (a lot of memory required). So to solve this we agreed to two things

1. We would have N+1 reports i.e. N reports for namespace specific events from n(variable) namespaces.
2. We would have some sort of pruning/trimming to make sure a report doesn’t grow too big

We started working on the first point by first creating two reports i.e. 1 policy report, with all the events that have namespace information in their output fields and 1 cluster policy report with all the other events. Next, to get N policy reports, while mapping we got the namespace string; for each namespace, we created/updated a report in that namespace itself.

For pruning, we simply kept a count variable that would hold the number of events in the report, as soon as it crossed the maximum number allowed we removed the first event in the []Results slice and added the new event.

To make this more efficient we created a struct with report and count:

struct defined in code that is used as a value in `polreports` map

and then we created a map[<namespace>]*<policyreport> , so every time we get an event with namespace information, we check whether we have a report for that namespace and if not we create a new one with a unique id to prevent multiple Falcosidekick instances updating the same report, and add it to the map:

code snippet showing how policy report is added after checking for namespace

Phase 4

Phase 4 concluded with adding documentation to both wg-policy-prototypes and Falcosidekick, and adding configuration options for the user, and finally cleaning up the code/refactoring the code. The config options included:

1. Enabled: to enable policy report output to create/update policy reports; the default value is false. We need to explicitly enable Falcosidekick outputs.
2. Kubeconfig: address to the configuration file (default- ~/.kube/config)
3. FailThreshold: events with priority above this threshold are mapped to Fail in PolicyReportSummary and “high” severity; rest are mapped to Warn in PolicyReportSummary (and severity “low” if event priority is below the threshold and “medium if it is equal to threshold” ). The default value is 4.
4. MaxEvents: this specifies the maximum number of events in any of the N+1 reports; once the events go above this number the report starts self-pruning/trimming depending on the next config option.
5. PruneByPriority: while pruning; by default, the events that came initially will be deleted (FIFO); by enabling this config the events that came initially of low priority are deleted before initial events of higher priority

struct with all available config options in types.go

Finally, after 3 months, we had a working Falco adapter READY to be merged to Falcosidekick!

Running Policyreport output for Falcosidekick

• To run the Kubernetes cluster locally, tools like kind or minikube can be used. Here are the steps to run the Falco adapter with a kind cluster.

1. kind create cluster --config=kind-config.yaml

(Check out an example of kind-config.yaml)

2. helm repo add falcosecurity https://falcosecurity.github.io/charts
3. helm repo update
4. kubectl create -f https://github.com/kubernetes-sigs/wg-policy-prototypes/raw/master/policy-report/crd/v1alpha2/wgpolicyk8s.io_clusterpolicyreports.yaml
5. kubectl create -f https://github.com/kubernetes-sigs/wg-policy-prototypes/raw/master/policy-report/crd/v1alpha2/wgpolicyk8s.io_policyreports.yaml
6. helm install falco falcosecurity/falco --set falcosidekick.enabled=true --set falcosidekick.policyreport.enabled=true falcosidekick.policyreport.kubeconfig=~/.kube/config falcosidekick.policyreport.failthreshold=3 falcosidekick.policyreport.maxevents=10

Above can be configured according to specifications

7. kubectl port-forward svc/falco-falcosidekick 2801

To get a Report summary

kubectl get clusterpolicyreports
kubectl get policyreports --all-namespaces

To get Reports

kubectl get clusterpolicyreports -o yaml 
kubectl get policyreports --all-namespaces -o yaml

To get reports in a separate YAML file you can use kubectl get clusterpolicyreports -o yaml > res.yaml or kubectl get policyreports --all-namespaces -o yaml > res.yaml

If the project is still under review when you read the article and you wish to try it out, check this document for the steps

What did I learn

This was my first big project and that too in an organization like CNCF, the journey alone taught me a million things one of them being tenacity is a great skill to have, I learned Golang by reading documentation, numerous blogs, and gophercises and most importantly learning by applying which was in contrast to how I would have learned a language at college i.e. theory then practical. I studied policy engines and CRDs and also learned and used cloud-native technologies like Kubernetes, Falco, Falcosidekick. It was a great experience to be a part of an organization and be a team player, meet deadlines, deal with stressful situations and other unexpected bumps along the road. Last but not the least, I have learned soft skills that proved to be very helpful as I could always get the right guidance when I asked and interacted with the community members.

With this, my journey as LFX Summer Mentee ’21 ends but my journey as an open source contributor will continue and grow, while remembering that we grow by uplifting others.

Thank you!