Systemtap for CoreOS Container Linux

Performance analysis in Linux is always an adventure, although it’s not my primary responsibility I do get the opportunity to investigate some exotic performance issues.

In this particular instance, I was looking into high rate of RX packets drops observed during saturation load test in our stress-test Kubernetes cluster (I’ll probably blog separately about the specifics of this issue and how we mitigated it).

Our Kubernetes clusters are deployed on top of CoreOS Container Linux and being a minimalistic Operating System for running containers workload it presents some challenges in terms of debugging/troubleshooting.

In most cases, the integrated Fedora based toolbox container will allow you to install and utilize any tool needed for further analysis, however for the sake of my analysis I needed to be able to utilize Systemtap.

For those unfamiliar, I highly recommend looking into it, on a very high level it is a tool that allows us to “tap” into the internals of the Linux Kernel and perform measurements and observations on how certain parts of it operate.

My particular use case was to measure the latency distribution of the net_rx_action kernel function which, as the name suggests, it handles the processing of incoming network packets.

The challenge?

Systemtap requires access to kernel’s debuginfo in order to be able to process the run-time information and tap into the relevant function calls and data to produce the required results.

Above is a challenge in the scope of CoreOS Container Linux since you can’t install anything in the host system itself (by design), it is possible to overcome this though I would avoid it — especially on production servers.

Using the available kernel debuginfo packages in the toolbox container is not an option as well since it won’t match the run-time kernel version.

Some Systemtap scripts may be rewritten to avoid the need of the kernel debuginfo packages but this requires more in-depth understanding of the underlying syscalls to find substitutions that will work and it won’t cover 100% of the use cases.

Quick and easy workaround?

After looking into various options, I was able to come up with a workaround (thanks to some helpful hints from CoreOS toolbox maintainers).

Here are the steps to reproduce:

• Connect to your toolbox container

$ toolbox

• Download and extract the build’s kernel, modules and sources (ensure you have enough free space).

$ source /media/root/etc/os-release; UNAMER=`uname -r| sed 's/-coreos//'`; wget -N http://builds.developer.core-os.net/boards/amd64-usr/${VERSION}/pkgs/sys-kernel/coreos-{kernel,modules,sources}-${UNAMER}.tbz2 && find . -maxdepth 1 -name 'coreos-*.tbz2' -exec tar --directory=/ -xvf '{}' \;

• [optional] This step may not be needed in your case, but you should compare the output of uname -v to the value of UTS_VERSION in:

/usr/lib/modules/`uname -r`/build/include/generated/compile.h

if they differ, update UTS_VERSION in the above file to match the output of uname -v.

• Download and build Systemtap from source, a good reference can be found here (dependencies can be fulfilled using dnf -y install <package(s)>). I recommend using the latest v4.0 of Systemtap.

With the above steps completed, you should have a fully functional Systemtap in your CoreOS Container Linux toolbox container.

Hope you’ll find this useful! thanks for tuning it and happy debugging :-)