HPy: Better Python C API in Practice

HPy is an alternative to the standard CPython C API for Python extensions developed jointly by GraalPy and PyPy developers. The main HPy design goal is to provide a simpler and future-proof API that abstracts and hides the implementation details of Python, such that, unlike the standard C API, it does not hinder improvements in CPython and works well with other alternative Pythons that may use different implementation strategies, such as a moving garbage collector instead of reference counting.

For Python users, HPy brings binary compatibility of Python extensions not only across different CPython versions, like the CPython stable API does, but also across different Python implementations. One binary release of an HPy based extension works with CPython, GraalPy, or PyPy. There is no need to recompile the extension from sources, nor to distribute dozens of binary releases.

Introduction

The “H” in HPy stands for “handle”. Using opaque handles instead of pointers to internal data structures, such as PyObject*, is the core concept in HPy.

However, do not expect that HPy completely turns the basic high-level concepts of the standard C API upside down. With HPy, you will still define extension methods, types, slots, and call API functions that, for example, convert a C long into a Python int object. The following code snippet shows two implementations of the same functionality. Although one uses the C API and the other the HPy API, they look very similar.

Now, you may be wondering: why should anyone want to port their existing Python extensions to HPy or choose HPy for their new Python extensions over the CPython C API? There are several distinguishing features and goals of HPy that derive naturally from its main design goal to provide a simpler API that abstracts and hides implementation details:

• Binary compatibility: one binary distribution of an HPy based extension will run on any CPython version, including future versions. What makes HPy different from the CPython stable ABI, is that the same binary will run also on any alternative Python implementation that natively supports HPy. At this point, those are GraalPy and PyPy.
• Superb performance on alternative Python implementations: with the CPython C API, GraalPy and PyPy have to emulate the implementation details of CPython, which comes at a substantial cost. HPy extensions perform significantly better on alternative Pythons.
• Raw CPython API performance: the universality of HPy may add a tiny bit of a performance overhead on CPython. To remove this overhead, HPy extensions can be compiled in the “CPython ABI” mode. In such case, all the HPy API calls will be rewired to CPython API calls at build time. The result will be an HPy independent standard CPython extension with all the drawbacks of the CPython API.
• Debug context: at extension loading time, one can choose to load an HPy extension with a so-called debug context. Note that this does not require re-compilation of the extension. The debug context intercepts all API calls and looks for common errors, especially handle leaks (similar to reference counting errors in CPython). Running your tests in a debug context, provided that they have good test coverage, should ensure that your extension will work well on any Python that implements the HPy specification correctly and that the extension is not unintentionally relying on some unspecified behavior.

While we are expecting to soon see a bright future where nearly every Python extension is based on HPy, this is unfortunately not the current state of affairs. Therefore another important design goal of HPy is to ensure a smooth migration path from the C API to HPy. What helps the Python extension developers in that regard?

• Similarity to the C API where possible: as mentioned above, the basic concepts in HPy API are similar to the existing C API: you still define extension functions, types, etc.
• Combining C API and HPy: the C API and HPy API can be combined within one Python extension. HPy provides functions that allow conversion between HPy and PyObject* such that one can mix HPy code with C API code. HPy modules can expose both "legacy" C API-based builtins and HPy builtins, which allow migration of the extension iteratively one function at a time. This is also the case for exposed types and type slots.

What makes the HPy API different from the C API? Some of the basic differences from the perspective of a Python extension developer are:

• HPy requires passing HPyContext* as the first argument to all API functions.
• HPy does not expose any internal structures. For example, the HPy C type is just an opaque struct, and it is up to the Python implementation to decide what are the actual contents. The user code cannot make any assumptions about it.
• HPy does not expose reference counting in the API. Instead, every single HPy handle must be explicitly closed. There is no reference "borrowing" or "stealing" known from the C API.

Other differences and more details about HPy API can be found in its documentation.

Kiwisolver and Matplotlib Case Studies

We believe that HPy can serve as an API for writing Python extensions that is sufficiently abstract and future proof while balancing the amount of effort necessary to port existing Python extensions to a new API. Because HPy effectively hides the implementation details of the underlying Python implementation, it allows alternative Pythons to realize their full potential while running Python native extensions and, moreover, HPy could allow CPython itself to experiment and embrace optimizations that would not be possible with the standard CPython API. One example we have already considered is passing unboxed integers in handles, which can speed up math operations on CPython.

To be more concrete, in follow up blogs we will describe some of the HPy features with an example of how did experimental ports of the Kiwisolver¹ and the well-known Matplotlib Python packages to HPy:

• Incremental porting process and HPy performance: Porting Matplotlib from C API to HPy
• HPy binary compatibility with Kiwisolver
• Debugging features of HPy with Kiwisolver

Stay tuned for those and other blog posts about GraalPy and HPy!

[1] Kiwisolver provides a Python binding for an efficient C++ implementation of the Cassowary constraint solving algorithm and it is a dependency of the Matplotlib package.