eBPF — From Scripts To Production

eBPF is probably one of the best features added to the Linux kernel this decade, allowing users to integrate with the kernel’s code without changing it or disrupting it. It has been around for a while now, but the industry only recently started to adopt it into their production environments.

There are a few reasons for this, one of those reasons is limitations of old eBPF verifiers and lack of features: It took time for the verifier to be intelligent enough to allow various eBPF functionalities and include newer eBPF features, which ultimately make this technology useful, and even more time until those kernels were used in leading cloud environments in the industry. In this series of posts we will discuss about some of those limitations and features, and describe how we overcome them.

Before we launched our eBPF program to a new environment, we encountered a problem in some of the test environments we set up. The eBPF program wouldn’t load and returned permissions denied error, which corresponds to the EACCESS error code. This is how the output looked like:

eBPF permission denied

We encounter the error “R1 type=map_value expected=fp”, which means that the value of R1 is of type map_value, and the verifier expected it to be fp, which means a stack pointer. The interesting thing was that it only happened on test environments with kernel version 4.9. Looking at the verifier’s code, we discovered the exact location our code was rejected:

https://elixir.bootlin.com/linux/v4.9/source/kernel/bpf/verifier.c#L961

The verifier makes sure the type of the argument equals to the expected type, which is PTR_TO_STACK. As we could anticipate, on a later version (kernel 4.18) this was fixed by allowing this exact use-case of PTR_TO_MAP_VALUE:

https://elixir.bootlin.com/linux/v4.18.20/source/kernel/bpf/verifier.c#L1983

How did we solve it on kernel 4.9?

Instead of calling the map helper function (bpf_perf_event_output in our case) with a pointer to the map value, we copied the variable from the map to a local variable on the stack, and then called the map function with local stack variable. If the variable is a struct that is too large to be allocated on the stack, we split it up to multiple structs of the same kind, but this approach cannot be applied to every use-case…

Interestingly, while writing this post I came across the specific commit that inserted this fix to the kernel: d71962f3e627 , and indeed, we can see it was merged on v4.18-rc1. Nevertheless, in my experience, some images with older kernels included a patch with this commit, for example, Amazon’s Linux 2 image with kernel version 4.14 allows this functionality as well.

We can learn a few take-aways from this situation:

1. Test your eBPF programs against multiple kernel versions.
   If you use a specific cloud provider, you should include in your tested kernel versions the default instance image, at the bare minimum.
2. Never be afraid to use the source code of the verifier to your advantage!
3. Don’t expect your eBPF code to work just because it worked many times before.

This is the first post in a series of posts about implementing eBPF programs in production environments, and the problems we have faced and encountered while doing so. Watch out for future posts…