We just got a new super-power! Runtime USDT comes to Linux

Mary Marchini
Oct 30, 2017 · 4 min read

A new tracing capability has arrived in Linux: the ability to instrument user-level statically defined tracing (USDT) probes defined in dynamic languages such as Python and Node. Prior Linux USDT support has been limited to the static code, such as C or C++. Now, USDT probes can be defined in a Node.js program and used from there, allowing new analysis tools that combine custom Node.js tracepoints along with other library and kernel events. Dtrace-capable operating systems have libusdt, and now Linux has libstapsdt — a library to create USDT probes at runtime. Both have a similar API, as libstapsdt is inspired on libusdt, but they work very different internally.

Libstapsdt uses the same structure provided by Systemtap SDT. Thus it should work with any tool able to trace SDT probes from Systemtap (for example, bcc’s tplist and trace tools). It is written in C, which makes it easy to be wrapped by many other languages. If you want to learn more about how libstapsdt works internally, you can look at our documentation here.

In this tutorial, we will show how to use eBPF/bcc to trace probes using Node and Python wrappers for libstapsdt. We assume a basic understanding of software instrumentation and tracing tools on this tutorial.

Installing Dependencies

To use either Node or Python wrappers, you first need to install libstapsdt on your system. On Ubuntu 16.04, you can install libstapsdt from a PPA. For other distributions, you can build it from source following the instructions here.

In this tutorial, we will use eBPF/bcc (version 0.4.0 or higher) to trace USDT probes. You can install the latest stable version by running:

Now you can either try the NodeJS Example or the Python Example.

NodeJS Example

You can install NodeJS’s wrapper by running:

Copy the following code and paste it into a file named index.js:

You can then run it with the following command. It will print “Trying to fire probe…” every 1 second.

You can now use some tools to trace your program. For example, if you run the command below in another terminal, it will print the arguments returned on line 26 every time it fires a probe. You can also see that now your Node application is printing “Probe fired!” in the terminal.

Now you can use any tool from bcc which uses USDT probes to analyze your Node programs with USDT probes!

This example is also available as an ASCIICast:

Python Example

You can install Python’s wrapper by running:

Copy the following code and paste it into a file named demo.py:

You can then run it with the following command. It will print “Trying to fire probe…” every 1 second.

You can now use some tools to trace your program. For example, if you run the command below in another terminal, it will print the arguments passed on line 18 every time it fires a probe. You can also see that now your Python application is printing “Probe fired!” in the terminal.

Now you can use any tool from bcc which uses USDT probes to analyze your Python programs with USDT probes!

This example is also available as an ASCIICast:

Where to go from here

Instrumentation on Linux is growing fast, especially with recent improvements to eBPF on the Kernel and the fantastic tools present on bcc. With libstapsdt, we’re taking another step by adding the ability to developers to instrument their code in dynamic languages.

There are still some refinements to be done on libstapsdt, but it works for most use cases. If you want to contribute to libstapsdt, you can find us on our Github page. If you decide to write a wrapper, please let us know (we’ll add it to the list of wrappers on the main page)!

Sthima Insights

Sthima has over 10 years of experience with development and…

Sthima Insights

Sthima has over 10 years of experience with development and setup of software in several companies in Brazil, helping professionals and companies in the process of choosing technologies, infrastructure and best practices in the projects.

Mary Marchini

Written by

Software architect and coffee lover. http://mmarchini.me/

Sthima Insights

Sthima has over 10 years of experience with development and setup of software in several companies in Brazil, helping professionals and companies in the process of choosing technologies, infrastructure and best practices in the projects.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store