Deepanshu Thakur
Aug 25, 2017 · 5 min read
  • A curiously recurring Python

In this post, I’ll try to explain the C++ CRTP idiom and how to bind it with python using pybind11.

Curiously Recurring Template Pattern or CRTP is an idiom in C++. It is a type of static or compile time polymorphism. The true power of CRTP is that it resolves all the virtual function calls at compile time thus avoids the extra memory of vPtr, vTableand also avoid the extra pointer dereference.

If a virtual routine is called many times (order of hundreds of thousands), it drops the performance of system, reason being each time the routine is called, its address needs to be resolved by looking through vTable using vPtr. Extra indirection (pointer dereference) for each call to a virtual method makes accessing vTable a costly operation and CRTP being able to resolve all these method calls at compile time which also avoids these costly operations and make a significant impact on the performance.

Now we’ll see an example of CRTP.

Above is a simple example of showing the usage of CRTP. Here you can see I have created a GenericParser. This Generic Parser is a template class and have 3 methods, inorderParser, callAppropriate and private function processNode. The in-order parser will parse the tree nodes in-order fashion.

in-order tree traversal

The processNode is a function, which does the processing on the nodes, in this case printing a message. The callAppropriate function is where the magic is happening. This method will statically cast the this pointer to the template type provided at the time of creation of the object, and call the processNode of the casted object.

Now here is the tricky part. The EmployeeParser is publicly derived from GenericParser and the GenericParser is a template class with template type of EmployeeParser

In the main function, we create a EmployeeParser object and invoke the inorderParser. The inorderParser is function of GenericParser. It will invoke the callAppropriate function. The callAppropriate will cast this object into type T and invoke T’s processNode function. In this example type T is actually EmployeeParser so eventually it is EmployeeParser's processNode function, that got invoked. Thus the polymorphism is achieved. See the code in action.

Now that we know what CRTP is, we can now jump unto the main point of this blog which is the python bindings with CRTP. To bind the C++ code with python I will be focusing on the pybind11.

Okay, so the thing is, you cannot bind the C++ CRTP with python in a “statically way”, because the CRTP depends on a template type that you might not know before compile time (that is, before binding it to python, the template type could be defined by the user of your code). In C++ whenever you write your main function and in that main you can provide the type of template at the time of object creation. In above example in main, I provided the template type as EmployeeParser to my GenericParser class. And after binding this class the user might want to initiate this EmployeeParser maybe with another C++ class say MilitaryParser

Let’s start out bindings with a simple example.

interface.h

I’ve created an Interface class. This is a template class. Then you get two classes impl_a.h and impl_b.h

And we can have a C++ main like below

When you execute above main, you will see P5ImplA and P5ImplB printed out, as expected.

Now we want to export this example into python. Meaning you want to export the interface to python and provide a python function to be able to “plug” one of the implementation to the interface, at “running time” (in a python interpreter, let’s say).

Before doing that, let’s see how to export ImplA to python, in a static way, so you can understand how I generalized this example. Because ImplA inherits from Interface<ImplA>, you have to export both to Python to be able to use ImplA. You might write something like a bindings.cpp file.

Now you’re able to use ImplA in python! Given the “make.common” file, “makefile” and the directory “pybind11/”, you just have to run make bindignsto be able to run a python interpreter which loads the module bindings and call ImplA().print() that will display P5ImplA.

Now let’s see how to generalize this example so we can dynamically export implementations of our interface.

First thing to notice is that we can generalise the way we’re exporting our Interface<ImplA>. Indeed, we could just write the bindings for Interface, using a template parameter for the implementation. So we could still export ImplA using this bindings.cpp

And now, the idea is to write a python function that will generate this part of the file automatically, replacing ImplA by the implementation name the user wants to use.

PYBIND11_MODULE(binding, m) {
register_interface<Interface<ImplA>>(m);
pybind11::class_<ImplA, Interface<ImplA>>(m, "ImplA")
.def(pybind11::init<>());
}

And please really note that your python function shouldn’t generate too much code. Indeed, since you write the bindings of your interface (e.g. your API) in a C++ file, using a template parameter when referencing an implementation, you don’t need your python function to generate again all this code to export your API (because your implementation inherits from the interface so it will already have all the API available). For example, I clearly exported the print function of our interface in the C++ function register_interface() but I didn’t repeat that when exporting ImplA (I just needed to export its constructor, that’s it).

write_pybind11_module writes our bindings for the implementation in a file bindings.cpp, depending on the type impl_t provided (ImplA, ImplB, etc.). Since we have to choose the right header to include (impl_a.h, impl_b.h, etc.) in this file, we need a little json config file that maps the type impl_t provided to the paths to the needed includes. Then we have to load this config file to retrieve the right includes and write that in C++ #include “impl_a.h”. That’s what load_includes is for.

Finally, we need a last python function to run write_pybind11_module, compile the code and provide the new generated shared library (our python bindings) to our current python environment! That’s what gen does. It runs write_pybind11_module to write the bindings.cpp file depending on the implementation type impl_t provided by the user (ImplA, ImplB, etc.) then compiles it by calling make in a new process and finally, it returns a new python module by calling importlib.import_module that takes the new generated shared library in parameter.

And here it is, we’re able to dynamically export and use our implementations of Interface, in Python: http://codepad.org/Ji3T3fKk (feel free to open a new python interpreter in the directory of my git repo and type these commands, it should work.)

Of course it’s a simple example but nonetheless, it’s the same strategy for exporting a real complex library that uses the CRTP idiom.

)
Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade