Linux Foundation Mentorship (LFX) project — Kubernetes Policy WG: KubeArmor support for Policy Report CRD
30th August 2021 — This is the exact date when an email in my inbox read that I was selected as a mentee for the Fall term of the Linux Foundation Mentorship (LFX) program and ask me for which project? A hint — It is an open-source application orchestrator (simple meaning — it orchestrates containerized cloud-native microservices apps.) Yes, I am talking about the most beloved Kubernetes project!
A snapshot of the most exciting email in my student life —
It is said that if you have the right people to mentor you and if you are a part of a thriving open-source community of experts, the practical knowledge you gain is going to immensely help you in your career. I had the privilege of having the best combination of the above 2 things. I am grateful to my mentors — Mritunjay Sharma and Jim Bugwadia who guided me at every stage of my mentorship period. Every time I had a doubt, error in my code, or any obstacle which was hindering my project work, I had people not only from the Kubernetes community but also from the KubeArmor community helping me tackle it. I would especially like to thank folks from the KubeArmor community — Barun Acharya, Rahul Jadhav, and Jaehyun Nam for their constant support and help in setting up KubeArmor on my machine, helping me understand KubeArmor-related code and a lot of other things. A huge thanks to Anushka Mittal (who worked on a similar project for Falco-sidekick) for her tips and tricks.
Alright, let’s deep dive into the project now!
The scope of the project was to build an external adapter that will convert outputs received from KubeArmor to a policy report and will create/update the report in accordance with the PolicyReport Custom Resource Definition (CRD). How about that for a sentence full of gibberish words? Let’s take some time to understand what each of the above terms mean.
First of all, what is KubeArmor?
KubeArmor is a cloud-native runtime security enforcement system that restricts the behaviour (such as process execution, file access, and networking operation) of containers and nodes at the system level. It allows operators to define security policies and apply them to Kubernetes. Then, KubeArmor will automatically detect the changes in security policies from Kubernetes and enforce them to the corresponding containers and nodes. If there are any violations against security policies, KubeArmor immediately generates alerts with container identities. If operators have any logging systems, it automatically sends the alerts to their systems as well.
The basic architectural overview of KubeArmor is given in the snapshot below
Next, what is a Custom Resource Definition (CRD)?
The Kubernetes CustomResourceDefinition API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify. The Kubernetes API serves and handles the storage of your custom resource.
But are we using the Kubernetes CRD here? Well, the answer is no. The Kubernetes Policy Working Group (WG) has already defined a Policy Report CRD that helps unify outputs from multiple policy engines (some examples of policy engines are KubeArmor, Kube-bench, and Falco). This helps cluster-admins with managing clusters as policy results can be viewed and managed easily as Kubernetes resources from any Kubernetes management tool (kubectl, dashboard, Octant, etc.)
Awesome, that should be enough information to get started. Nearly two weeks in the first month of my mentorship period went into research and creating a proper roadmap to make sure we set our project in the right direction. The first task I wanted to accomplish was to get KubeArmor up and running on my machine. The guide sounded tedious at first and do note — KubeArmor operates only with Linux security modules (LSMs), meaning that it can work on top of any Linux platforms (such as Alpine, Ubuntu, and Container-optimized OS from Google) if Linux security modules (e.g., AppArmor, SELinux, or BPF-LSM) are enabled in the Linux Kernel. KubeArmor will use the appropriate LSMs to enforce the required policies. Meaning, I had to use a Virtual Box on my MacOS to run KubeArmor. Virtualization itself is a tedious task on a mac. Thanks to Rahul from KubeArmor who suggested that I use the vagrant environment documented in KubeArmor’s development guide. Finally, I had KubeArmor set up on my machine and got the alerts to print in the console.
The thing which is important to us here is the term “alerts” from KubeArmor. Whenever a policy violation is triggered, KubeArmor generates an alert (in JSON format) which looks like below —
{
"Timestamp":1631622914, "UpdatedTime":"2021-09-14T12:35:14.580230Z", "ClusterName":"Default", "HostName":"kubearmor-dev", "NamespaceName":"multiubuntu", "PodName":"ubuntu-1-deployment-5d6b975744-rrkhh","ContainerID":"67f0a1cdf83f9899ce2d4d77e9e81b220b49b2a2c29e7007dedac6b611b01c5f", "ContainerName":"ubuntu-1-container", "HostPID":32322, "PPID":29888, "PID":100, "PolicyName":"ksp-group-1-proc-path-block", "Severity":"5", "Message":"block the sleep command", "Type":"MatchedPolicy", "Source":"bash", "Operation":"Process", "Resource":"/bin/sleep 1", "Data":"syscall=SYS_EXECVE", "Action":"Block", "Result":"Permission denied"}
Neat, we have an alert generated in the JSON format. Looks good, but how do we get control over it? For this, we wanted the struct of the kubearmor output where these fields are defined. If there was no such pre-defined struct, we would have to create a struct of our own. But thankfully, (and special thanks to folks from the KubeArmor community) we had this struct already defined in the protobuf package in KubeArmor. The next step in the project was to figure out an approach through which we will be able to interact with the Policy Report CRD. And again, thanks to Mritunjay for already generating the client-codecs for our CRD using the Kubernetes code-generator. The generated code had to be generated only once and now it can be reused in any number of external adapters to create policy reports :)
The next important part was to map the above JSON fields to our CRD fields. Wait, but what is mapping exactly? To put it in simple words- The fields in our Policy Report CRD are different from the fields defined in the struct that is responsible for kubearmor outputs. The CRD fields need to be assigned values from the output (KubeArmor alerts) provided by the policy engine (KubeArmor). Initial discussion of mapping was done in this doc. Below is a snippet of how mapping was implemented in code-
return &v1alpha2.PolicyReportResult {
Source: PolicyReportSource,
Policy: Alert.PolicyName,
Scored: false,
Severity: v1alpha2.PolicyResultSeverity(sev),
Result: "fail",
Description: Alert.Message,
Category: Alert.Type,
Properties: map[string]string {
"updated_time": Alert.UpdatedTime,
"cluster_name": Alert.ClusterName,
"host_name": Alert.HostName,
"namespace_name": Alert.NamespaceName,
"pod_name": Alert.PodName,
"container_id": Alert.ContainerID,
"container_name": Alert.ContainerName,
"host_pid": strconv.Itoa(int(Alert.HostPID)),
"ppid": strconv.Itoa(int(Alert.PPID)),
"pid": strconv.Itoa(int(Alert.PID)),
"tags": Alert.Tags,
"source": Alert.Source,
"operation": Alert.Operation,
"resource": Alert.Resource,
"data": Alert.Data,
"action": Alert.Action,
"result": Alert.Result,
}Next, we set up a gRPC client to get the alerts stream from KubeArmor. The implementation of the code can be found out here — https://github.com/haardikdharma10/kubearmor-adapter/blob/main/main.go
And finally, we were able to create our very first kubearmor-policy-report with the ability to only create and not update. Since our aim here was to create and update namespace-specific policy reports, this means that if a policy report is already existing in that particular namespace, it gets updated and if we are unable to find an already existing report in that namespace, we’ll create a new report with the specified name. The implementation of this logic can be found here.
The last 2 weeks were for testing, adding documentation on how to use the adapter, and refactoring of code. That done, the KubeArmor adapter was ready to be merged into the wg-policy-prototypes repository.
Steps to create our first policy report :
# 1. Clone the kubearmor-adapter GitHub repository
git clone https://github.com/haardikdharma10/kubearmor-adapter
# 2. cd into kubearmor-adapter directory
cd kubearmor-adapter
# 3. Create a CustomResourceDefinition
kubectl create -f crd/v1alpha2/wgpolicyk8s.io_policyreports.yaml
# 4. Run main.go program
go run main.go
Note: Make sure you have the KubeArmor service running in the background before you go run main.go. If not, you can cd into KubeArmor/KubeArmor and run make clean && make run.
# 5. Open a new terminal window and deploy the multiubuntu microservice.cd KubeArmor/examples/multiubuntu
kubectl apply -f .#6. Deploy a security policy for testingcd KubeArmor/examples/multiubuntu/security-policies
kubectl -n multiubuntu apply -f ksp-group-1-proc-path-block.yaml#7. Trigger a policy violation by running the following commandkubectl -n [namespace-name] exec -it [pod-name] -- bash -c "/bin/sleep 1"
Note: In this example, namespace-name is multiubuntu and you can get the pod name by running kubectl get pods -n multiubuntu. An example pod-name is ubuntu-1-deployment-5d6b975744-rrkhh.
Once this command is executed, you'll get the output as below in the terminal window where main.go is running:
If you can see the output as above, this means that your first policyreport is created. You can now stop running the main program.
Viewing reports
You can view the summary of the created policyreport by running the following command:
kubectl get policyreports -n [namespace-name]To view the policyreport in yaml format, you can use:
kubectl get policyreports -n multiubuntu -o yamlTo view the report in a separate yaml file you can use:
kubectl get policyreports -o yaml > res.yamlA new file res.yaml will be created in the kubearmor-adapter directory. You can view it by running cat res.yaml.
To delete the policyreport, you can use:
kubectl delete policyreports -n [namespace-name] [policy-report-name]In our example, namespace-name is multiubuntu and policy-report-name is kubearmor-policy-report.
What did I learn?
The journey is more fruitful than the destination. These 3 months were full of learning new things, figuring out stuff, reading documentation, meeting deadlines, and tackling unexpected obstacles that came along the way. Learning Golang was fun. The resources I used to learn Golang were- This YouTube tutorial by freeCodeCamp, playing with https://gobyexample.com/, and the Go Documentation came in handy. Kubernetes ecosystem can be really overwhelming especially for beginners and I would highly recommend reading The Kubernetes Book by Nigel Poulton and Pushkar Joglekar which explains all the k8s related concepts in a very simple language. I learned about CRD’s from the Kubernetes CRD documentation itself which I think is the quickest way to get your head around CRD’s. Last but not the least, this opportunity helped me become a better communicator, a better team player, more timezone friendly, and of course, a better open-source contributor :)
If you have any comments/feedback/suggestions or need any help related to the project, feel free to reach out to me on Kubernetes/CNCF/KubeArmor slack. You can also reach out to me on my Twitter if that’s convenient :) Thanks for reading!
