Integrating C++ and Python for Scientific Computing

Generated by DALL·E 2 with the prompt: “Create a minimalistic banner for a tutorial on integrating C++ and Python”.

Scientific computing has been revolutionizing research and development, with its impact felt across data science, physics, engineering, biology, and finance. At the heart of this revolution is the synergy between different programming languages, particularly C++ and Python. This tutorial isn’t just about theory; it’s about unlocking a new realm of possibilities by combining Python’s simplicity and C++’s speed.

In the realm of programming, C++ and Python stand as giants. Python, known for its user-friendly syntax, vast libraries, and strong community support, is often the language of choice for scientists and researchers for data analysis, machine learning, visualization, and rapid prototyping. However, when it comes to performance-intensive computations, C++ emerges as a preferred option due to its optimized memory management and execution speed. Imagine performing complex data analyses with Python’s ease and visualizing results with the raw power of C++ — this is what we aim to achieve here.

But what happens when you merge the robustness of C++ with the versatility of Python? You get a combination that can tackle complex computing challenges with both speed and ease. This is especially vital in research and industry, where the need for efficient yet rapid development is paramount.

Integrating C++ and Python isn’t just a theoretical exercise; it’s a practical need in software development. To achieve this, several libraries are at your disposal, each with its unique strengths and limitations. Here we just mention a few:

1. Boost.Python (https://www.boost.org/doc/libs/1_84_0/libs/python/doc/html/index.html): A well-documented and widely used library that offers seamless interoperability between C++ and Python.
2. SWIG (https://www.swig.org/): Great for multi-language projects, but can be less efficient and less Pythonic.
3. pybind11 (https://pybind11.readthedocs.io/en/stable/): Stands out for its modern approach, being lightweight and easy to use.
4. Cython (https://cython.org/): Known for allowing C extensions in Python-like syntax.
5. ctypes (https://docs.python.org/3/library/ctypes.html): A part of Python’s standard library, suitable for basic tasks involving C functions.

Among these, pybind11 shines for its balance of power and simplicity. It’s a header-only library, meaning it’s easy to include in your projects without heavy setup.

Why pybind11?

pybind11 is not just another library; it’s a modern tool that resonates with the demands of today’s industry. It’s designed for simplicity, yet it doesn’t compromise on power. By offering more Pythonic bindings compared to other alternatives, it becomes an ideal choice for a wide range of projects.

pybind11 is all about creating Python bindings of existing C++ code. Its design is akin to Boost.Python but focuses more on being minimalistic. By leveraging C++11 features, it sidesteps the complexities associated with its competitors, making it a go-to for modern C++/Python integration.

Prerequisites

To effectively use pybind11 for integrating C++ and Python, there are several prerequisites that you should be familiar with.

1. Basic knowledge of Python and C++: this includes familiarity with Python’s syntax, data types, and basic programming constructs, as well as C++ syntax, object-oriented programming, and memory management.
2. C++ compiler: A C++ compiler that is compatible with the C++11 standard or later is required, as pybind11 utilizes C++11 features. Common compilers like GCC (https://gcc.gnu.org/), Clang (https://clang.llvm.org/), or MSVC (https://visualstudio.microsoft.com/vs/features/cplusplus/) (on Windows) are suitable.
3. Python environment: You should have Python (https://www.python.org/) installed on your system. Most Python distributions come with the necessary tools and libraries to get started. It’s also beneficial, although not mandatory, to be familiar with virtual environments in Python for managing dependencies. One such example is Conda (https://docs.conda.io/en/latest/).
4. Development tools:
   – Build tools: Knowledge of build systems like CMake (https://cmake.org) is important, as pybind11 is commonly integrated with CMake for building extensions. This is not essential for the basics covered in this tutorial.
   – IDE/Text editor: While not strictly necessary, using an Integrated Development Environment (IDE) or a text editor that supports both Python and C++ can make your development process smoother.

Setting the stage

Getting started with pybind11 is straightforward. You have multiple options depending on your environment:

• Via PyPI: pip install pybind11.
• Using conda: conda install -c conda-forge pybind11.
• Using Homebrew/Linuxbrew: brew install pybind11.
• Git submodule (https://pybind11.readthedocs.io/en/stable/installing.html#include-as-a-submodule): Ideal for Git-based projects, where you can include pybind11 as part of your repository.
• Installing from source.

For those interested in exploring the code examples and concepts discussed in this post in greater detail, the entire codebase is available on this GitHub repository: https://github.com/pcafrica/introduction_to_pybind11. This availability ensures reproducibility and allows readers to experiment directly with the code, facilitating a hands-on learning experience. You can clone or fork the repository to try out the examples, make modifications, and see the effects in real-time on your machine.

Before diving into coding examples, it’s crucial to set up our environment correctly. For all our examples, we’ll include the main pybind11 header and define a convenient namespace shortcut:

#include <pybind11/pybind11.h>
namespace py = pybind11;

This setup allows us to access pybind11 features quickly, ensuring our code remains clean and readable.

Binding 101: creating simple functions

Imagine you have a simple C++ function, like one that adds two numbers. How do you make this function accessible in Python? That’s where pybind11 shines. Here’s a glimpse of how you’d do it:

int add(int i, int j) {
    return i + j;
}

// Syntax: PYBIND11_MODULE(<module name>, <module object>)
PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin"; // Optional module docstring.
    m.def("add", &add, "A function that adds two numbers"); // Bind the function and include it in the module.
}

The magic behind the PYBIND11_MODULE macro is the heart of pybind11's binding mechanism. It creates a function that Python calls upon importing the module.

More specifically, the PYBIND11_MODULE() macro is designed to define a function that is executed when Python issues an import command. It takes two input arguments. The name of the module, in this case example, is provided as the first argument of this macro and should not be enclosed in quotes. Following this, the second argument, labeled m, instantiates an object of type py::module_. This variable acts as the pybind11 placeholder structure for binding functions, classes, variables, documentation, and much more into the module.

Through the use of the py::module_::def() method, binding code is produced, effectively allowing the C++ function add() to be accessible and usable within Python. Finally, the py::module_::doc() method allows to include a descriptive string that explains what the function does, its parameters, return type, and any other relevant information. This documentation is then accessible in Python, typically through the built-in help() function or by using the .__doc__ attribute.

Keyword and default arguments

Pybind11 also allows you to define keyword arguments, enhancing the readability of your Python function calls. For example:

// m is the pybind11 module object.
m.def("add", &add, "A function which adds two numbers",
      py::arg("i"), py::arg("j"));

In this example, py::arg("i") and py::arg("j") explicitly define the arguments i and j for the add function. This modification lets Python users call your function using keyword arguments, like add(i=3, j=4) or add(j=3, i=4). This approach is clearer and more self-documenting, especially when a function accepts several parameters. It helps to avoid confusion about the order of arguments and makes the code more self-explanatory, enhancing the overall readability and maintainability of the code.

In addition to defining keyword arguments, Pybind11 also supports the specification of default arguments, which further enhances the flexibility and user-friendliness of your Python-bound C++ functions. Default arguments allow you to specify default values for parameters, making them optional when the function is called from Python. For example:

// m is the pybind11 module object.
m.def("add", &add, "A function which adds two numbers",
      py::arg("i") = 1, py::arg("j") = 2);

In this case, py::arg("i") = 1 and py::arg("j") = 2 set default values for i and j. When this function is called from Python without arguments, it uses these default values. So, a call like add() in Python is equivalent to add(1, 2), and both add(3) and add(j=4) are valid calls, using the default for the non-specified argument(s).

Compiling

Notice how little code was needed to expose our function to Python: all details regarding the function’s parameters and return value were automatically inferred using template metaprogramming.

pybind11 is a header-only library, hence it is not necessary to link against any special libraries and there are no intermediate (magic) translation steps. Given a C++ compiler, the above example, assumed to be saved in a example.cpp source file, can be compiled into a module using the following command:

c++ \                                            # The name of the compiler.
  -std=c++11 -O3 -shared -fPIC \                 # Specify the standard C++11, enable optimizations and generate a shared library.
  $(python3 -m pybind11 --includes) \            # Add flags for including pybind11 directory.
  example.cpp \                                  # The file to be compiled.
  -o example$(python3-config --extension-suffix) # Output filename, with proper Python extension suffix.

The command above will generate a dynamic module example.cpython-version-arch.so that can be imported in Python as follows:

import example

help(example) # Print module docstring.
example.add(1, 2); # Output: 3

Supported data types

pybind11 offers comprehensive support for a wide range of data types, facilitating seamless conversions between C++ and Python, which is crucial for effective integration. Among the supported types are the standard Python data types such as integers, floating-point numbers, strings, lists, tuples, dictionaries, and sets. These are automatically converted to and from their C++ counterparts, like int, double, std::string, std::vector, std::tuple, std::map, and std::set, respectively. This automatic conversion simplifies the process of passing data between the two languages.

A notable feature of pybind11 is its compatibility with the NumPy (https://numpy.org/) library, a cornerstone in Python’s scientific computing stack. NumPy arrays are efficiently converted to and from C++’s Eigen (https://eigen.tuxfamily.org/index.php) library types or STL containers like std::vector and std::array, allowing for high-performance matrix and vector operations crucial in scientific computations. This integration means that NumPy arrays can be directly passed to C++ functions expecting Eigen types without the need for explicit conversion, and vice versa. Crucially, this seamless interoperability extends to other powerful libraries built on top of NumPy, such as SciPy (https://scipy.org/), Matplotlib (https://matplotlib.org/), pandas (https://pandas.pydata.org/), and many more.

A large number of data types are supported out of the box and can be used seamlessly as functions arguments, return values or with py::cast in general. For a full overview, see the conversion table (https://pybind11.readthedocs.io/en/stable/advanced/cast/overview.html#conversion-table) and the general documentation (https://pybind11.readthedocs.io/en/stable/advanced/cast/index.html).

C++ vs. native Python: a performance showdown

Let’s take a common scientific computing task: dense matrix multiplication. Python makes it easy, but when performance is key, C++ takes the lead. Here’s how pybind11 bridges this gap.

Consider the following streamlined C++ code, which represents a matrix using a vector of vectors. We assume the matrices mat1 and mat2 are compatible in size for multiplication:

#include <pybind11/numpy.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

#include <vector>

std::vector<std::vector<double>>
matrix_multiply(const std::vector<std::vector<double>> &mat1,
                const std::vector<std::vector<double>> &mat2) {
  const size_t rows = mat1.size();
  const size_t cols = mat2[0].size();
  const size_t inner_dim = mat2.size();

  std::vector<std::vector<double>> result(rows, std::vector<double>(cols, 0));

  for (size_t i = 0; i < rows; ++i) {
    for (size_t j = 0; j < cols; ++j) {
      for (size_t k = 0; k < inner_dim; ++k) {
        result[i][j] += mat1[i][k] * mat2[k][j];
      }
    }
  }

  return result;
}

// pybind11 module definition.
namespace py = pybind11;
PYBIND11_MODULE(matrix_ops, m) {
  m.def("matrix_multiply", &matrix_multiply,
        "A function which multiplies two NumPy matrices");
}

Compile this C++ code into a Python module with the command:

c++ \
  -std=c++11 -O3 -shared -fPIC \
  $(python3 -m pybind11 --includes) \
  matrix_multiplication.cpp \
  -o matrix_ops$(python3-config --extension-suffix)

In Python, define an equivalent function to compare performance:

def matrix_multiply_python(mat1, mat2):
    rows = len(mat1)
    cols = len(mat2[0])
    inner_dim = len(mat2)
    
    result = [[0 for _ in range(cols)] for _ in range(rows)]
    for i in range(rows):
        for j in range(cols):
            for k in range(inner_dim):
                result[i][j] += mat1[i][k] * mat2[k][j]
    return result

Next, generate two random square matrices of size n = 500 using NumPy for testing:

import numpy as np

# Create random matrices.
n = 500
mat1 = np.random.rand(n, n)
mat2 = np.random.rand(n, n)

Now, let’s benchmark both the C++ and Python implementations:

import matrix_ops
import time

# Benchmark C++ implementation (using pybind11).
start_time = time.time()
result_cpp = matrix_ops.matrix_multiply(mat1, mat2)
cpp_duration = time.time() - start_time
print(f"C++ implementation took {cpp_duration:.4f} seconds.")

# Benchmark Python implementation.
start_time = time.time()
result_python = matrix_multiply_python(mat1, mat2)
python_duration = time.time() - start_time
print(f"Python implementation took {python_duration:.4f} seconds.")
print(f"Speedup: {python_duration // cpp_duration}.")

On an Intel® Core™ i7–8565U CPU @ 1.80GHz the C++ implementation takes on average around 0.15 seconds, whereas native Python takes up to around 60 seconds. This equates to a speedup of approximately 300 to 400 times!

The impact of such huge difference becomes even more dramatic if we consider algorithms where matrix-matrix products have to be computed several times.

This example illustrates how a computationally intensive task, when implemented in C++ and exposed to Python using pybind11, can lead to significant performance gains, all while maintaining the ease of use that Python offers.

It’s important to note that the matrix multiplication implementations in our example, both in Python and C++, are not optimized in any specific manner, particularly regarding cache utilization and efficiency.

As an engaging exercise for the reader, it would be insightful to compare the performance of the C++ and Python implementations against native NumPy matrix multiplication, i.e., using the @ multiplication operator (e.g., mat1 @ mat2). NumPy is highly optimized for vectorized operations, making it an excellent benchmark for performance comparison.

Conclusions

With pybind11, the fusion of C++ and Python in scientific computing isn’t just seamless: it’s powerful!

As we’ve seen, whether it’s simple function binding or complex matrix operations, pybind11 stands as a bridge, allowing you to leverage the strengths of both languages. Remember, this is just the beginning. The real adventure starts when you apply these concepts to your unique challenges and data types.

Dive in, experiment, and witness how integrating C++ and Python can transform your scientific computing projects!

References and further readings

• Examples discussed in this post: https://github.com/pcafrica/introduction_to_pybind11
• pybind11 documentation: https://pybind11.readthedocs.io/en/stable/
• Example pybind11 module built with a CMake-based build system: https://github.com/pybind/cmake_example
• Example pybind11 module built with a Python-based build system: https://github.com/pybind/python_example
• pybind11 testsuite: https://github.com/pybind/pybind11/tree/master/tests