Custom Raspberry Pi Debug Kernel

4 min readJul 29, 2018

or how to mess with your kernel.

Introduction

This is a description on how to cross compile[1] a custom Linux kernel for the Raspberry Pi 3B including kernel debugging and memory leak investigation configurations.

[1] Using ahost Fedora 28 to a guestRaspberry Pi system; instructions may differ depending on your OS, for example for Debian-based OS’s it can be something similar to gcc-arm-linux-gnueabihf

Setting Up

Getting the kernel

First thing to do while reading this, is to start downloading the kernel repo; It can take some time.

git clone --depth=1 https://github.com/raspberrypi/linux/

Current branch at the time of writing (July 2018) is the rpi-4.14.y.

Using an old `.`config

If there’s already a known working kernel configuration file on hand from the OS then it’s advised to use it. It usually is on the location /proc/config.gz. For the file to exist, the option Kernel .config support should be enabled in the kernel. This is the case on the current Raspberry Pi 3 and in case you don’t see the file there then do a sudo modprobe configs and the file will magically appear in the right place.

# On the host
KERNEL=kernel7
# Transfer your old file from your Pi via scp
zcat /proc/config.gz > .config
# Clean any old config files
make mrproper
# make the kernel aware of using an old configuration
make ARCH=arm CROSS_COMPILE=arm-linux-gnu- oldconfig
# and setup with new options; look on the next section
make ARCH=arm CROSS_COMPILE=arm-linux-gnu- menuconfig

Detailed debug info configuration

As a reference here’s the (missing) required configuration for a debug and memory leak detection kernel. For more information consider reading the Linux Kernel Documentation. This is for reference only, the .config file shouldn’t be manually edited as there are dependencies between the options; that may result in a non/malfunctioning kernel.

# Note: The commented options are important but they were already enabled
# = CONFIG_PRINTK_TIME=y
# = CONFIG_DEBUG_KERNEL=y
# = CONFIG_KALLSYMS=y
# = CONFIG_KGDB=y
# = CONFIG_KGDB_SERIAL_CONSOLE=y
# = CONFIG_KGDB_KDB=y
# = CONFIG_PROC_FS=y
# = CONFIG_MAGIC_SYSRQ = y
# = CONFIG_SYSFS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_PREEMPT=y # CLASH WITH VOLUNTAR?Y
CONFIG_KDB_KEYBOARD=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_KCORE=y
CONFIG_CRASH_DUMP=y
CONFIG_DEBUG_INFO=y
CONFIG_KEXEC=y
CONFIG_SLUB_DEBUG_ON=y
CONFIG_SLUB_STATS=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_KMEMLEAK=y
CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=4096
CONFIG_DEBUG_KMEMLEAK_TEST=m

Making the kernel

Fire it and grab a cup of tea while waiting for the compilation!

make -j6 ARCH=arm CROSS_COMPILE=arm-linux-gnu- zImage modules dtbs

Debug Notes

Add to the /boot/cmdline.txtthe following, whereuser_debug=Nis one of the following:

1 — undefine instruction events
2 — system calls
4 — invalid data aborts
8 — SIGSEGV faults
16 — SIGBUS faults

Installing onto the SD Card

To identify the SD run lsblk before and after the SD insertion. You ‘re going to get two partitions: p1 should be the boot partition, while p2 the rootfs. Now create two folders in oder to mount the fs, for example:

mkdir -p mnt/bootfs
mkdir mnt/rootfs

Mount and install the modules:

sudo mount /dev/xxxp1 mnt/bootfs
sudo mount /dev/xxxp2 mnt/rootfssudo make ARCH=arm CROSS_COMPILE=arm-linux-gnu- INSTALL_MOD_PATH=mnt/rootfs modules_install

After that you need to copy the kernel and device tree blobs onto the SD card:

# Run all that as root
cp mnt/bootfs/$KERNEL.img mnt/bootfs/$KERNEL-backup.img
cp arch/arm/boot/zImage mnt/bootfs/$KERNEL.img
cp arch/arm/boot/dts/*.dtb mnt/bootfs/
cp arch/arm/boot/dts/overlays/*.dtb* mnt/bootfs/overlays/
cp arch/arm/boot/dts/overlays/README mnt/bootfs/overlays/
umount mnt/bootfs
umount mnt/rootfs

Debugging

This is just a reference and some things to try using the compiled kernel to get more debugging information. I am writing a new article that is addressing the issue in more detail. Stay tuned!

Memory leaks

# Using kernel memory leak debugging
# sudo mount -t debugfs nodev /sys/kernel/debug # superfluous — already mounted
sudo bash -c “echo scan > /sys/kernel/debug/kmemleak”
sudo cat /sys/kernel/debug/kmemleak # Repeat to see updated info# IF the `echo scan…` part above *fails*, check /var/log/kern.log
# What I saw: “Kernel memory leak detector disabled”,
# “Early log buffer exceeded (####), please increase DEBUG_KMEMLEAK_EARLY_LOG_SIZE”# I increased CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE and rebuilt/reinstalled the kernel# Another handy command:
sudo slabtop -sc

Oops and bugs

If the kernel is compiled with the option CONFIG_KALLSYMS then the dump is usable as it is. The most important information to keep is the EIP line ignoring the 0010:. By looking at the uncompressed kernel (vmlinux and not zImage or vmlinuz) with nm vmlinux | sort | less one can extract the name list, match it against the EIP from the kernel crash and look up the kernel function that contains the offending address.

Note: The address can’t be matched exactly (so it cant’ be grepped) but it’s a good starting point as to discover the location of the call.

Alternatively, you can use gdb on a running kernel. (read-only; i.e. you cannot change values or set break points.) To do this, first compile the kernel with -g; edit arch/arm/Makefile appropriately, then do a make clean. You’ll also need to enable CONFIG_PROC_FS (via make config).

After you’ve rebooted with the new kernel, launch gdb vmlinux /proc/kcore. You can now use all the usual gdb commands. The command to look up the point where your system crashed is l *0xXXXXXXXX. (Replace the XXXes with the EIP value.)

gdb’ing a non-running kernel currently fails because gdb (wrongly) disregards the starting offset for which the kernel is compiled.

Troubleshooting

Can’t load kernel modules

Check which modules are loading through the conf file emacs -nw /etc/modules-load.d/*.conf (try commenting a few out) and run sudo systemctl status systemd-modules-load.service which should give more detailed information on what is failing.

Cannot find wifi card / No Internet

It seems a bit weird as sometimes after a kernel installation the internet installation seems to fail. I suggest that before installing the modified kernel do a sudo raspi-update to get the current firmware blobs.

Useful articles

Many of the information above has been compiled through a lot of experimentation, crashed systems, dumped cores and a lot of reburned SD cards. Thanks all the authors for their Free and Open Source approach.

Linux Kernel Admin Guide
Debugging the linux kernel
The kernel newbie debugging
Debug kernel space memory leak
Trace and debug the linux kernel
Newbie corner kernel
Kernel debugging tips