Quick peek into kernel land keyboard events handling with ftrace and dynamic_debug

Gao Zhiyuan
Dec 19, 2018 · 3 min read

This post briefly introduces some internals about linux kernel, especially in relation to event device handling and keyboard event tracing. We shall use ftrace and dynamic_debug to dive inside.

Your keyboard devices are listed under /dev/input/eventX and ‘X’ might subject to change. You can use evtest /dev/input/eventX to tell the value of ‘X’. In my case, I found my keyboard at /dev/input/event4, since I found corresponding device description with evtest.

First we would like to figure out which kernel function handles /dev/input/event4 and where it copies buffer from to user space.

To peek into the kernel land, we would like to find a kernel name from the /proc filesystem. We can use tail -f /dev/input/event4 and ps aux | grep tail to find the pid of this process.

Then cat /proc/14844/stack would allow us to read kernel stack backstraces of this process.

The top function is where in kernel code we are busy looping right now. And from here we can start tracing kernel source code by finding its callers.

You can browse kernel source from bootlin or download kernel source code from github. evdev_read is used through a function pointer and it is used by evdev_handler. Using ctags and cscope evdev_handler is used by evdev_event called by evdev_events.

From here we can use ftrace to find a more complete calling graph.

cd /sys/kernel/debug/tracing
echo "evdev_events" > set_ftrace_filter
echo 1 > tracing_on # start the trace
cat available_tracers
echo function > current_tracer # we use function tracer
echo 1 > options/func_stack_trace
:> trace
cat trace

I’m experimenting on a qemu vm and function tracer gives me below results and it might be different on your computer.

=> evdev_events
=> input_to_handler
=> input_pass_values.part.8
=> input_handle_event
=> input_event
=> atkbd_interrupt
=> serio_interrupt
=> i8042_interrupt
=> __handle_irq_event_percpu
=> handle_irq_event
=> handle_edge_irq
=> handle_irq
=> do_IRQ
=> ret_from_irq
=> native_safe_halt
=> default_idle
=> do_idle
=> cpu_startup_entry
=> start_kernel
=> secondary_startup_64

It seems __handle_irq_event_percpu and its callers are not specificly relevant to keyboard events. Tracing i8042_interrupt, serio_interrupt and atkbd_interrupt, i8042_interrupt handles interrupts and sends incoming bytes to upper layers. The incoming bytes are processed atkbd_interrupt and the function uses dev_dbg(&serio->dev, "Received %02x flags %02x\n", data, flags);.

dev_dbg is widely used by linux kernel for debugging needs. Steps to use it:

echo "file atkbd.c +p" > /sys/kernel/debug/dynamic_debug/control
cat dynamic_debug/control |grep atkbd.c
dmesg -C
dmesg -w

And now dmesg would emit keycodes kernel has received at every key event.



lwn: The dynamic debugging interface

lwn: Debugging the kernel using Ftrace

kernel.org: Dynamic debug howto

Gao Zhiyuan

Written by

Open Source proselytizer, openSUSE Advocate

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade