Using Python with C++. Part1.

Using Python and C++ together using Pybind library.

Summary

In this story we are going to implement a simple c++ function that will be called from a python script.

A bit of background

In 2019 i began to use Python for various reasons: first of all because i wanted to put my hands on TensorFlow and make my first experiments with machine learning. Then, because, respect to C++, it was very easy to use. It allows in short time to prototype code, create scripts, etc. The power of Python is due basically to the enormous amount of free libraries available such as NumPy, Pandas, Scipy, Librosa, and, last but not least TensorFlow and Pytorch.

At certain point i needed to run some python code with classes i’ve already implemented in c++. Not only. Sometimes i needed to improve performance using c++ optimized code.

In fact, a recurrent story that many python and c++ programmers have listened to is about the old myth: Python is easy but slow, C++ is difficult but is fast.

This could be true in some cases but, as most of you will probably know, when you need to go fast, python can be as fast as C++. This because many python libraries have main function calls written in C++ under the hood.

So, the question is: How is it possible? Can we take the best of both worlds? Why don’t we take the simplicity of Python and the speed of C++?

The answer to these questions is: Yes, it can be done !

(dutiful mention from the movie Frankstein junior (italian translation of “It can be done”, a little different from the original american version “It could work!” but makes the idea clearer of what we will be able to do :)

Good, but how ?

In order to establish a communication between Python and C++, we must create some special bridges between the two languages. In other words, we must create the so called Python bindings to use C++ from Python and Python from C++.

Language Bindings

Ok, but what are Language Bindings? Bindings are wrapper libraries that bridge two programming languages, so that a library written for one language can be used in another language.

One library that allows to create python binding is PyBind11. Why did i choose this one ? For four reasons in particular:

• Easy to use
• Well documented
• Header only library (you don’t have to to link against any other libraries)
• Lightweight (few kB)

First Steps and installation

First, you need to download the library from github:

git clone https://github.com/pybind/pybind11.git 

Project Setup

Let’s setup everything we need to compiling / running the code:

C++ Tools

I personally use Visual Studio Code with C/C++ extension and CMake extension installed to edit, debug and run c++ code, but also to create and edit CMake files.

Python Tools

• A Python 3.6+ environment should be fine.
• PyCharm CE or other ide to test python code.

Folder Structure

So, i created a project structure like this:

• Inside the newly created Pybind11Example folder i put:
• A CMakeLists.txt file.
• A source folder with my custom hpp and cpp files.
• A third party library folder where to put pybind11 lib and other libraries you need.

The cpp file

The source file example.cpp found in the source/example folder contains the methods, classes, structs, … that we want to expose to Python.

In this case we are going to implement an add method. In order to make it visible to Python we use the macro PYBIND11_MODULE(<module_name>, <m>). You just need to pass two arguments to macro:

• The module name example without quotes. This is the library name it will used in python with the instruction import example.
• A variable of type py::module_ m. The last instruction is telling the compiler to bind the AddCpp method (defined here in the cpp) with the method add we are going to use in Python.

#include <pybind11/pybind11.h>

float AddCpp(float a, float b)
{
    return a + b;
}

PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin"; // optional module docstring
    m.def("add", &AddCpp, "A function that adds two numbers");
}

The CMakeLists.txt file

In this basic example we are setting the project name, project source directory, and other options.

• The CMake command add_subdirectory will import the pybind11 project.
• The pybind11_add_module function command will take care of all the details needed to build a Python extension module on any platform. It basically will add a library target called <name> to be built from the listed source files (example.cpp)

cmake_minimum_required(VERSION 3.16)
project(example)
set(CMAKE_CXX_STANDARD 17)
set(SOURCE "${PROJECT_SOURCE_DIR}/source/example")
set(THIRDPARTYLIB "${PROJECT_SOURCE_DIR}/ThirdPartyLib")

# pybind11
add_subdirectory(${THIRDPARTYLIB}/pybind11)

# module
pybind11_add_module (
    example 
    ${SOURCE}/example.cpp
)

• PyBind11 will require a Python installation. CMake will find the current installed python library and will use it to build the source module files.

From the terminal (inside the Pybind11Example folder) you have to launch these commands:

mkdir build
cd build
cmake ../
cmake --build .

• This will generate a library file named example.cpython-37m-darwin.so
• Note that the postfix cpython-37m-darwin is telling you that is a library made with Python 3.7 on a MacOs operating system. Depending from the Python version and from the operating system this will change a bit.

You should have an output similar to this one:

-- The C compiler identification is AppleClang 13.0.0.13000029
-- The CXX compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- pybind11 v2.11.0 dev1
-- Found PythonInterp: /Users/stefano/opt/anaconda3/envs/TF2Projects/bin/python (found suitable version "3.7.9", minimum required is "3.6") 
-- Found PythonLibs: /Users/stefano/opt/anaconda3/envs/TF2Projects/lib/libpython3.7m.dylib
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Performing Test HAS_FLTO_THIN
-- Performing Test HAS_FLTO_THIN - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/stefano/Desktop/Pybind11Example/build

[ 50%] Building CXX object CMakeFiles/example.dir/source/example/example.cpp.o
[100%] Linking CXX shared module example.cpython-37m-darwin.so
[100%] Built target example

Working with Python

So, now, let’s create a simple python test file: AddTest.py

• Move the file example.cpython-37m-darwin.so into the folder that contains AddTest.py

import example

sum = example.add(4,5)
print(sum)
9

As you can see you just need to import the module example and call example.add as you would for any other python function.

This ends the first part of this journey. In the next part we are going to do a more complex example.

See you next time, and Happy Reading :)
Stefano