Using pybind11 for Python Bindings of C++ code (Linux/WSL)

Update: Comprehensive Guide to Pybind11

The complete and in-depth documentation for Pybind11 is readily accessible online here.

This guide offers comprehensive coverage of the topic of C++ Python binding, serving as an invaluable resource for anyone interested in this subject matter.

The Pybind11 library is capable of mapping a wide range of core C++ features to Python, facilitating the integration of these two powerful programming languages. The mapped features include but are not limited to:

1. Functions: Pybind11 can handle functions that accept and return custom data structures, whether per value, reference, or pointer.
2. Methods: Instance and static methods are well within the capabilities of Pybind11.
3. Overloaded Functions: The library can easily manage overloaded functions.
4. Attributes: Both instance attributes and static attributes can be handled.
5. Exception Types: Pybind11 can deal with arbitrary exception types.
6. Enumerations: The library supports the mapping of enumerations.
7. Callbacks: Pybind11 facilitates the implementation of callbacks.
8. Iterators and Ranges: The library allows the integration of iterators and ranges.
9. Custom Operators: Pybind11 supports custom operators.
10. Inheritance: The library supports single and multiple inheritance.
11. STL Data Structures: Pybind11 can map STL data structures.
12. Smart Pointers: The library supports smart pointers with reference counting, such as std::shared_ptr.
13. Internal References: Pybind11 can manage internal references with accurate reference counting.
14. C++ Classes: The library allows C++ classes with virtual (and pure virtual) methods to be extended in Python.

For practical application and a deeper understanding of these features, consult the examples provided in the official documentation. Each situation is well-illustrated, offering readers hands-on knowledge of how to use Pybind11 effectively.

Happy learning, and best of luck in your journey with C++ Python binding!

Introduction:

Python Bindings allow us to take advantage of C/C++ libraries from Python. Hence, Python Bindings are helpful when:

➤ We already have a stable and tested library written in C/C++, and we want to use it in Python.

➤ We create an application where the processing speed is crucial since C/C++ is faster than Python.

I first encountered the need to do Python Bindings last year when I was working on a real-time application and realized I needed some stable functions that I had already written in C++.

For this tutorial, I will work with pybind11. My first attempt when starting to work with Python Bindings was with Cython, but the main disadvantage for me was I had to modify the syntax of my original C++ code and mix it with Python syntax (“Cython is Python with C data types”).

Cython, on the other hand, helped me to achieve a higher processing speed.

Summary of what is coming next:

Before starting our journey to pybind11, I want to ensure that we are all on the same page.
For this tutorial, I will use:
➤ Visual Studio Code
➤ Windows Subsystem for Linux (WSL)
➤ CMake
For the environment setup in Visual Studio, as well as how to use a CMake file, please check these links, especially if you are new to the concepts:

Get Started with C++ and Windows Subsystem for Linux in Visual Studio Code

Get started with CMake Tools on Linux

Steps:

In the Linux terminal, go to the home directory and then to the user directory (in my case, it’s called “bianca”).

cd /home/bianca

Now we are in the “~” directory.
Here create a new folder called PythonBindings.

mkdir PythonBindings
cd PythonBindings

Now open Visual Studio Code to create our file by typing:

code .

Let’s create our main.cpp file where our function of interest will be located. At first, I will start with a straightforward function that adds two numbers (ints or floats).

main.cpp file:
#include <pybind11/pybind11.h>
namespace py = pybind11;
double sum(double a, double b) {
   return a + b;
}
PYBIND11_MODULE(SumFunction, var) {
   var.doc() = "pybind11 example module";
   var.def("sum", &sum, "This function adds two input numbers");
}

As you can see, I created the function “sum” that adds two numbers of type “double.” I also included the header “#include <pybind11/pybind11.h>”. This, for now, should highlight in vscode that there is an issue with this header, but we will get to that in a moment.

The end part is the PYBIND11_MODULE macro and is the core that will help us with the binding. You can find more information here:

First steps - pybind11 documentation

Now, let’s create the CMake file. Go in vscode to View → Command Palette → CMake: Quick Start → Enter a name for our project: Projectpybind11 → Create an executable.
The default CMake file should look like this:

CMakeLists.txt file:
cmake_minimum_required(VERSION 3.0.0)
project(Projectpybind11 VERSION 0.1.0)
include(CTest)
enable_testing()
add_executable(Projectpybind11 main.cpp)
set(CPACK_PROJECT_NAME ${PROJECT_NAME})
set(CPACK_PROJECT_VERSION ${PROJECT_VERSION})
include(CPack)

But if you go to the main.cpp file, the same error will be present since we haven’t installed pybind11 yet!

Back in our PythonBindings folder, we will clone pybind11 from the official git repository:

git clone https://github.com/pybind/pybind11.git

Now, an installation is required:

cd /home/bianca/PythonBindigs
python3 -m pip install pytest numpy scipy
sudo apt install -y cmake python3-dev libeigen3-dev libboost-dev git
cd pybind11
cmake -DDOWNLOAD_CATCH=1
mkdir build
cd build
cmake ..
sudo make install
cd ..

We also have to modify the CMake file to look like this:

CMakeLists.txt file:
cmake_minimum_required(VERSION 3.0.0)
project(Projectpybind11 VERSION 0.1.0)
include(CTest)
enable_testing()
add_subdirectory(pybind11)
pybind11_add_module(SumFunction main.cpp)
set(CPACK_PROJECT_NAME ${PROJECT_NAME})
set(CPACK_PROJECT_VERSION ${PROJECT_VERSION})
include(CPack)

Back in the PythonBindings folder we will type the following commands:

cd /home/bianca/PythonBindigs
mkdir build
cd build
cmake ..
make
cd ..
touch main.py

In the /home/bianca/PythonBindigs/build we will see the SumFunction.cpython-38-x86_64-linux-gnu.so file which is a shared library in Linux.

Now, in the main.py file write:

from build.SumFunction import *
print(sum(3,2))

Finally:

cd build
cmake .. && make && python3 ../main.py

This should output our desired result: 2+3 = 5. Try this example with floats. Example: 2.3 + 5.2 = 7.5

Note: In vscode the highlighting error for “#include <pybind11/pybind11.h>” should not be present anymore, but if it is, vscode might ask you to configure your project → click yes → everything should look good now (this happened to me). But, even though the code was highlighted with red, the commands in the terminal gave the correct result.

This is the end of Using pybind11 for Python Bindings of C++ code (Linux/WSL) — Part I.

Thank you for reading!
This is my first post here. I went through some tutorials before being able to use pybind11 properly, so I decided to present to you how I managed to do it.

Feel free to leave a comment if you have any questions or suggestions!

Link to GitHub repository:

GitHub - biancainsk/Pybind11-Tutorial