We just got a new super-power! Runtime USDT comes to Linux

A new tracing capability has arrived in Linux: the ability to instrument user-level statically defined tracing (USDT) probes defined in dynamic languages such as Python and Node. Prior Linux USDT support has been limited to the static code, such as C or C++. Now, USDT probes can be defined in a Node.js program and used from there, allowing new analysis tools that combine custom Node.js tracepoints along with other library and kernel events. Dtrace-capable operating systems have libusdt, and now Linux has libstapsdt — a library to create USDT probes at runtime. Both have a similar API, as libstapsdt is inspired on libusdt, but they work very different internally.

Libstapsdt uses the same structure provided by Systemtap SDT. Thus it should work with any tool able to trace SDT probes from Systemtap (for example, bcc’s tplist and trace tools). It is written in C, which makes it easy to be wrapped by many other languages. If you want to learn more about how libstapsdt works internally, you can look at our documentation here.

In this tutorial, we will show how to use eBPF/bcc to trace probes using Node and Python wrappers for libstapsdt. We assume a basic understanding of software instrumentation and tracing tools on this tutorial.

Installing Dependencies

To use either Node or Python wrappers, you first need to install libstapsdt on your system. On Ubuntu 16.04, you can install libstapsdt from a PPA. For other distributions, you can build it from source following the instructions here.

sudo add-apt-repository ppa:sthima/oss
sudo apt-get update
sudo apt-get install libstapsdt0 libstapsdt-dev

In this tutorial, we will use eBPF/bcc (version 0.4.0 or higher) to trace USDT probes. You can install the latest stable version by running:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys D4284CDD
echo "deb https://repo.iovisor.org/apt/xenial xenial main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

Now you can either try the NodeJS Example or the Python Example.

NodeJS Example

You can install NodeJS’s wrapper by running:

npm install usdt

Copy the following code and paste it into a file named index.js:

.75

You can then run it with the following command. It will print “Trying to fire probe…” every 1 second.

node index.js

You can now use some tools to trace your program. For example, if you run the command below in another terminal, it will print the arguments returned on line 26 every time it fires a probe. You can also see that now your Node application is printing “Probe fired!” in the terminal.

# PID is the pid number for "node index.js"
sudo /usr/share/bcc/tools/trace -p [PID] 'u::firstProbe "%d - %s", arg1, arg2'

Now you can use any tool from bcc which uses USDT probes to analyze your Node programs with USDT probes!

This example is also available as an ASCIICast:

libstapsdt NodeJS Wrapper + eBPF/bcc trace

Python Example

You can install Python’s wrapper by running:

pip install stapsdt

Copy the following code and paste it into a file named demo.py:

You can then run it with the following command. It will print “Trying to fire probe…” every 1 second.

python demo.py

You can now use some tools to trace your program. For example, if you run the command below in another terminal, it will print the arguments passed on line 18 every time it fires a probe. You can also see that now your Python application is printing “Probe fired!” in the terminal.

# PID is the pid number for "python demo.py"
sudo /usr/share/bcc/tools/trace -p [PID]'u::firstProbe "%s - %d", arg1, arg2'

Now you can use any tool from bcc which uses USDT probes to analyze your Python programs with USDT probes!

This example is also available as an ASCIICast:

libstapsdt Python Wrapper + eBPF/bcc trace

Where to go from here

Instrumentation on Linux is growing fast, especially with recent improvements to eBPF on the Kernel and the fantastic tools present on bcc. With libstapsdt, we’re taking another step by adding the ability to developers to instrument their code in dynamic languages.

There are still some refinements to be done on libstapsdt, but it works for most use cases. If you want to contribute to libstapsdt, you can find us on our Github page. If you decide to write a wrapper, please let us know (we’ll add it to the list of wrappers on the main page)!