HPy: Debugging Features with Kiwisolver

In the previous post of our HPy series we took a look at how HPy delivers binary compatibility across multiple CPython versions, as well as multiple Python implementations, such as GraalPy. This was demonstrated using the Kiwisolver Python package. In this post we’ll use the Kiwisolver package HPy port again, this time to demonstrate another great feature of HPy: its debugging facilities.

As mentioned in the previous post, the migration of the Kiwisolver source code to the HPy API was similar to the experience of migrating Matplotlib. However, one notable point is that the original Kiwisolver uses the cppy C++ library, which provides the Resource Acquisition is Initialization pattern (RAII) for PyObject reference counting.

For the initial Kiwisolver HPy port, we removed the usages of cppy and implemented the lifetime management of HPy handles manually. Manual management of resources is an error-prone task and this allowed us to extensively exercise the HPy debug context feature.

Note: it is perfectly possible to provide similar C++ wrappers for HPy API, but at this point this is left for future work.

Once all the Kiwisolver unit tests passed on the HPy port, we turned on the HPy debugging features by wrapping each test with a LeakDetector context manager. This can be illustrated with the following pseudocode.

On entry, the LeakDetector remembers all currently open HPy handles and other HPy-related resources and on exit it checks that there are no new open HPy handles or other resources, i.e., that run_the_test() properly cleaned-up after itself.

In practice, we create a pytest fixture in our conftest.py to do that:

HPy exposes a pytest fixture called hpy_debug, which activates LeakDetectorjust as we did here in our fixture and in the future may activate additional HPy debugging facilities.

The functionality of the LeakDetector class is activated only if the HPy extension is loaded with HPy debug context. We can do that by exporting the environment variable HPY_DEBUG=1.

Moreover, the debug context feature works only if the extension is built in the HPy “universal” mode. It cannot work in the HPy “CPython ABI” mode, because all the calls to HPy API are rewired to call the corresponding CPython API at compile time and we cannot change that at runtime. We can check in which mode we are loading our extension by exporting the environment variable HPY_LOG=1.

When we were done with the first phase of porting to HPy, and before turning on the HPy debug features, the result of running the Kiwisolver tests produced this output:

Now let us turn on the debugging features:

Ooops! Looks like we still have some work to do. Better to find out now, than after a release! The segfault comes from an inconsistent internal state, which can always happen when things go too wrong in C. Let us start with the first test class (the output is shortened):

We are told to try hpy.debug.set_handle_stack_trace_limit, so let's do that in conftest.py. What we can see now, is, for example, this stack trace:

This HPy feature uses glibc’s backtrace and backtrace_symbols functions. There are various ways to get them to print line numbers and more useful information. However, looking at the stack traces, we can also recognize the C++ mangled name at the end of our shortened snippet that contains "BinaryAdd" and "Variable". This must be this C++ template method:

The mangled name of the function of the previous frame in the stack trace contains “BinaryMul”. So, we must be leaking the handle temp returned by the BinaryMul()(...) call. Indeed, the temp handle is not returned as the result, nor is there an HPy_Close(ctx, temp) call. The fixed method looks like this:

Rinse and repeat a few more times and the tests will finally pass.

Debug mode for CPython API?

You may ask: can something similar be implemented for PyObject* when using the CPython API? This is where the design of HPy that decouples it from reference counting and that decouples HPy handles from the Python objects themselves comes to play.

Every HPy API call that returns an HPy handle returns a fresh handle that needs to be closed by the caller. That handle is unique. While there may be several handles referring to the same Python object, they are all disconnected. When one of them leaks, we can pinpoint the exact location where that handle was created. With reference counting, on the other hand, if the count is not in balance, we have to inspect all of the places that incremented it and may have forgotten to decrement it. Which one of those locations is to blame cannot be determined by anyone but a human, because there is not a clear definition when a decref must follow incref that some automated tool could use.

Conclusion

This blog post described the HPy debug context feature, which augments HPy API calls with additional checks that discover potential bugs in API usage. Activation of the HPy debug context feature does not require recompilation of the extension.

More concretely, we demonstrated the HPy LeakDetector Python class that enabled us to discover HPy handle leaks. HPy handle leak is an issue similar to reference counting issues on CPython API. However, with HPy every native reference to a Python object is a distinct HPy handle, which helps to significantly narrow down the search for the root cause of the leak.

Get started with GraalPy here: www.graalvm.org/python