My LFX journey with KubeArmor — Improving system test coverage and practices

LFX Mentorship

Linux Foundation Mentorship Program (LFX Mentorship) is an open source program by the Linux Foundation that aims to promote the open source culture and community.

For more info regarding LFX mentorship, you can check out the official website — https://lfx.linuxfoundation.org/tools/mentorship/

I was fortunate to be a part of this awesome program in 2024 term 2. I applied for the organization KubeArmor under the project “Improve system test coverage and practices for KubeArmor”.

KubeArmor

KubeArmor is a runtime Kubernetes security engine. It uses eBPF and Linux Security Modules(LSM) for fortifying workloads based on Cloud Containers, IoT/Edge, and 5G networks. It enforces policy-based controls.

GitHub - kubearmor/KubeArmor: Runtime Security Enforcement System. Workload hardening/sandboxing…

Now let’s come to what I worked on and how it improved KubeArmor. My project aimed to solve the following issues:

1. Calculate the code coverage for KubeArmor running in all operational modes.
2. Writing host security policy tests for KubeArmor running in K8s mode and non-k8s mode.
3. Write container security policies for AppArmor and BPFLSM enforcers in unorchestrated mode.
4. Test KubeArmor in ARM machines and write automated workflows.
5. Automating testing in frequently used environments.

Calculating the code coverage of KubeArmor

Out of the above points, the first was the most complex one (although it doesn’t seems like when you read it!). This is because Golang’s testing frameworks don’t have a straightforward way to calculate coverage for E2E tests, which was the case in KubeArmor.

This was solved by creating a test binary of KubeArmor with coverage flags and then running the E2E tests against this test binary, so now at the end we get the coverage file which can then use to calculate coverage from Codecov or go tool cover .

These blogs shared by my mentors helped me a lot in understanding how this would work and how to implement it:

1. https://barun.cc/posts/cover-run/
2. https://blog.cloudflare.com/go-coverage-with-external-tests/
3. https://www.cyphar.com/blog/post/20170412-golang-integration-coverage

For context, KubeArmor can run with AppArmor or BPFLSM to enforce policies and can be installed in various environments, such as:

1. Kubenetes mode — to protect pods and nodes
2. Systemd mode — to protect linux hosts and containers
3. Container mode — as docker containers to protect hosts and other containers

So, now the task was to do automated testing of KubeArmor in all the above modes using GitHub Actions, generate their corresponding coverage files and merge them in the workflow.

This flowchart by my mentors made things simpler for me:

Merge coverage files from different runs

Host Security Policy (HSP) tests for KubeArmor

KubeArmor can protect hosts in Kubernetes mode by adding the flag -enableKubeArmorHostPolicy in the kubearmor daemonset.

In Systemd mode, host protection is enabled out of the box with the debian and rpm package.

My task was to write the HSP gingko tests for these 2 modes. ginkgo is a BDD testing framework for Golang and is used by KubeArmor to write the E2E tests.

These are the PRs that introduced the HSP tests for KubeArmor — #1857 and #1858.

Container Security Policy Tests

KubeArmor can enforce policies on containers also, but the functionality differs between AppArmor and BPFLSM.

In BPFLSM, everything works as usual and you can enforce policies on running containers easily.

But in AppArmor, Docker has this limitation where an AppArmor profile must exist and the container must be run with it specified as an option (with the —security-opt flag) in order for enforcement to work. More info here.

So, I had to write test suites for these 2 separate environments as we test both AppArmor and BPFLSM in GitHub Actions.

The conclusion

Overall, the mentorship was a great success for me. I learned a lot about runtime security and how KubeArmor leverages eBPF and LSMs to enforce policies. I got hands on some new testing frameworks like Ginkgo and some important features of GitHub Action workflows and automated testing.

Acknowledgements

I would like to extend my heartfelt thanks to Barun Acharya and Rudraksh Pareek for their invaluable support throughout my mentorship. Their guidance, especially during moments when I was stuck, was instrumental in helping me understand the complexities of the project. Their patience and willingness to explain challenging concepts made this journey not only smoother but also highly enriching. I’m incredibly grateful for their mentorship and the knowledge they imparted to me.

About Me

I love contributing to open source!

• LinkedIn — https://www.linkedin.com/in/navin-chandra-b610b2144/
• GitHub — https://github.com/navin772

Thank you for reading through my LFX Mentorship experience, feel free to contact me for any questions!