Open sourcing ptracer, a syscall-tracing library for Python

Elvis Pranskevichus, Pinterest engineer, Core Experience

Making Pinterest faster and more reliable is a constant focus for our engineering team, and using hardware resources more efficiently is a major part of this effort. Improving efficiency and reliability requires good diagnostic tools. That’s why today we’re announcing our newest tracing tool, ptracer, which provides granular syscall tracing in Python programs. In this post we’ll cover background on Pinterest’s codebase, why we needed a better tracer and how ptracer can help solve certain engineering problems.


However, using multiple processes has a price to pay–increased memory usage. In a large codebase, just the memory allocated for Python code objects and strings can reach into hundreds of megabytes, with a similar amount needed for application data. The total memory usage across several dozen workers can quickly get out of hand.

Notably, a large portion of data across workers is the same, like the Python code objects and other static bits. The solution to decrease memory usage is to make the worker processes share that data. Thankfully, there’s a way to do just that.

In POSIX systems, new processes are created by invoking the fork() system call which creates an almost identical copy of the calling process. Most importantly, the memory of the process isn’t physically copied right away. Instead, all memory pages shared between the parent and the child processes are marked with the special copy-on-write bit which triggers a copy only when a memory page is modified by one of the processes.

Image for post
Image for post
Figure 1. Memory sharing in forked processes

The worker processes start with almost perfect memory sharing but will gradually diverge.

The main source of memory divergence in Python programs is reference counting. In CPython, every Python object is represented by a structure which includes a reference counter that’s incremented whenever a reference to the object is made. Thus, in Python, just accessing an object is enough to trigger the memory page copy.

Similarly, the CPython garbage collector maintains a special record in every tracked Python object and will modify that record when a collection run is made. There are remedies to the latter problem, and Python 3.7 will provide a standard solution in the form of a gc.freeze() call.

Despite these limitations, a large proportion of memory can stay shared, leading to significant overall savings.

From supervisord to fork-server

Besides memory, the child process created by fork() also shares all open file descriptors with its parent and siblings. A file descriptor includes the current offset within the file, and sharing it between processes leads to interleaved reads and writes (which most programs will not expect and fail).

The usual solution to this problem is to immediately close all file descriptors (except stdin, stdout and stderr) after fork() returns in the child process. In our case this wouldn’t work, because the code that opened the file might rely on it to continue being open. Instead, we have to ensure no files or sockets are left open when we call fork() to spawn the workers.

Manually finding all places that open files without closing them is a daunting task. In a large codebase like ours it’s downright impossible, especially considering numerous third-party modules we depend on. Furthermore, fixing this issue once isn’t enough. We need to make sure new code doesn’t introduce regressions.

The only workable solution to this problem is to automate the checks and incorporate them into the regression test suite. To do this we need a mechanism to trace the program and intercept all system calls that create new file descriptors.

Many systems, including Linux, have the ptrace() system call, which allows tracing and interrupting the execution of other processes. Debuggers like gdb use it as well as the venerable strace tool. But we couldn’t simply use strace. While it does report the fact that a certain file was opened or closed, it doesn’t show where in the program this happened which makes finding the spot unnecessarily hard.

That’s why we built our own tracer–ptracer–to solve this problem.


Under the hood, ptracer uses a combination of threads and subprocesses to make the above possible.

Image for post
Image for post
Figure 2. ptracer architecture

When a ptracer context is entered, ptracer spawns a new thread in which the syscall callback will be executed, in addition to another thread used to extract a Python traceback from a given thread. It also spawns a subprocess which does the actual tracing.

Whenever a tracing subprocess detects a system call being made, it checks the syscall against the specified filter (if there is any). If the system call matches the filter, the traceback thread is called to extract the call stack, which is attached to the syscall record and put into the queue. The callback is called for each item in the syscall queue.

Solving the problem of open files pre-fork()

Future work

We welcome any issues, feedback and pull requests. For all our open source projects, check out our GitHub.

Acknowledgements: I’d like to thank Yury Selivanov, Jon Parise, Sam Meder and Joe Gordon for their help and feedback on the project.

Pinterest Engineering Blog

Inventive engineers building the first visual discovery…

Pinterest Engineering

Written by | Inventive engineers building the first visual discovery engine |

Pinterest Engineering Blog

Inventive engineers building the first visual discovery engine, 200 billion ideas and counting.

Pinterest Engineering

Written by | Inventive engineers building the first visual discovery engine |

Pinterest Engineering Blog

Inventive engineers building the first visual discovery engine, 200 billion ideas and counting.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store