How to Mix Rust and Python in Your Project

In the 2021 Stack Overflow Annual Developer Survey [1], Rust is ranked as the “most loved language” (for the sixth year!) and Python “as the most wanted” one. So why not blend the two languages! This introduction will set up a mixed project combining Python and Rust code.

Why? Python interpreted approach and dynamic typing accelerate our development workflows. But with more complex jobs, we can often reach performance issues.

Rust is a good candidate for enabling native performance upgrades of our Python projects. Native optimization uses low-level language and compiler to bypass the Python interpreter.

By combining Rust and Python, you get the best of both worlds: a fast and interactive development environment with Python and native performances with Rust.

Python + Rust Project — Image by Author

Why Two Languages in the Same Project?

In the following sections, I will set up an example of a mixed Python/Rust project. Our project will combine Python code and Rust within the same project and artifact.

The benefits of a mixed project are clear: you share your Rust and Python code in the same repository and create a single artifact. As a result, you combine the two languages in the same project; it becomes easier to iterate and extend your Python code with native Rust components. In addition, with only one artifact to deploy, it becomes easier to maintain and share your project.

The downsides are obvious: two languages in the same directory, how to build, test and deploy this kind of project?

How? PyO3 + Maturin

The primary way to integrate Rust and Python is the PyO3 Framework. With PyO3, we can write native Python modules in Rust. Of course, the framework also enables calling Python from Rust, but I will focus only on extending Python with a native Rust module.

PyO3 wraps your Rust code into a native python module. As a result, bindings generation is easy and fully transparent.

The tricky part is your code’s packaging when building mixed Python and native code projects (Python + [Rust, C, or C++]). So how to make wheels integrating Python and native code?

Python code is distributed without compilation and is platform-independent; installing the wheels creates .pyc files on the fly (Python bytecode). But our Rust code needs to be compiled and distributed as a shared library (binary code).

Packaging is the hard part of this project: create a generic, multi-platform way to generate hybrid Python-Rust packages.

A tool that can help you with this purpose is Maturin. Maturin manages creating, building, packaging, and distributing a mixed Python/Rust project.

PyO3 + Maturin — Image by Author

Initialize a Hybrid Project with Maturin

First, install Maturin Python package with pip.

$ pip install maturin

It includes the maturin binary, a command-line interface.

$ maturin --help
  
maturin 0.12.9
Build and publish crates with pyo3, rust-cpython and cffi bindings as well as rust binaries as
python packages
  
USAGE:
    maturin <SUBCOMMAND>
  
OPTIONS:
    -h, --help       Print help information
    -V, --version    Print version information
  
SUBCOMMANDS:
    build          Build the crate into python packages
    develop        Installs the crate as module in the current virtualenv
    help           Print this message or the help of the given subcommand(s)
    init           Create a new cargo project in an existing directory
    list-python    Searches and lists the available python installations
    new            Create a new cargo project
    publish        Build and publish the crate as python packages to pypi
    sdist          Build only a source distribution (sdist) without compiling
    upload         Uploads python packages to pypi

The maturin executable proposes several options. Let's zoom in on build and develop.

• build: Compiles Rust and integrate it with the Python package. It produces a wheel package mixing the binary artifact produced by Rust with the final Python code.
• develop: Useful while developing and debugging your project. This command builds and installs the newly created shared library directly in your Python module.

To test a mixed Rust-Python project, we can now initialize our library with the maturin new command.

$ maturin new --help
  
maturin-new
Create a new cargo project
  
USAGE:
    maturin new [OPTIONS] <PATH>
  
ARGS:
    <PATH>    Project path
  
OPTIONS:
    -b, --bindings <BINDINGS>    Which kind of bindings to use [possible values: pyo3, rust-cpython,
                                 cffi, bin]
    -h, --help                   Print help information
        --mixed                  Use mixed Rust/Python project layout
        --name <NAME>            Set the resulting package name, defaults to the directory name

For our project, we initialize a mixed Python/Rust project with PyO3 bindings by using the options --bindings pyo3 and --mixed my_project. The my_project argument matches the target project directory of this example.

$ maturin new --bindings pyo3 --mixed my_project

The generated project integrates a Python package in the directory my_project with a Rust project definitions Cargo.toml and a Rust based src directory.

$ tree my_project
  
my_project
├── Cargo.toml
├── my_project
│   └── __init__.py
├── pyproject.toml
├── src
│   └── lib.rs
└── test
    └── test.py
  
3 directories, 5 files

Ok, we have the basic project skeleton. We can now add a simple Rust function to expose.

Project Structure — Image by Author

Wrapping a Python Code with PyO3

We need to declare and export our Rust functions in a Python module callable by the Python runtime. We start by creating a module using the #[pymodule] Rust macro. In our function, my_project we will declare our bindings between Rust and Python.

→ rust «my_project/src/lib.rs» =
  
use pyo3::prelude::*; (1)
  
(2)
<<functions>>
  
#[pymodule]
fn my_project(_py: Python, m: &PyModule) -> PyResult<()> {
    (3)
    <<function_declarations>>
    Ok(())
}
  
(4)
<<tests>>

• (1) We include the Py03 definitions and macros.
• (2) In the <<functions>> block, we declare our Rust functions.
• (3) In the <<function_declarations>> block, we expose our Rust functions in the final Python module.
• (4) In the <<tests>> block, we add Rust unit test functions.

A Simple Function

Let’s start with a simple function for our <<functions>> block. Our function to export is_prime checks the primality of its input by dividing it by preceding numbers. For a number num, we check the division remainder of the numbers between 2 and √num.

→ rust «functions» =
  
#[pyfunction] (1)
fn is_prime(num: u32) -> bool {
    match num {
        0 | 1 => false,
        _ => {
            let limit = (num as f32).sqrt() as u32; (2)

            (2..=limit).any(|i| num % i == 0) == false (3)
        }
    }
}

• (1) The Rust macro #[pyfunction] generates code for Python binding.
• (2) Compute the upper bound of our trial division series.
• (3) Generate the trials and test the remainder of the division. 2..=limit generates a range between 2 and limit (included). any check if one of the generated elements satisfies the predicate.

In the <<function_declarations>> block, we add our function to our exported module.

→ rust «function_declarations» =
  
m.add_function(wrap_pyfunction!(is_prime, m)?)?;

In the <<tests>> block, we add a few simple unit tests.

→ rust «tests» =
  
#[cfg(test)]
mod tests {
    use super::*;
  
    #[test]
    fn simple_test_false() {
        assert_eq!(is_prime(0), false);
        assert_eq!(is_prime(1), false);
        assert_eq!(is_prime(12), false)
    }
  
    #[test]
    fn simple_test_true() {
        assert_eq!(is_prime(2), true);
        assert_eq!(is_prime(3), true);
        assert_eq!(is_prime(41), true)
    }
}

Build and run your Python module

With Maturin, building a native Rust module and exporting it in the Python interpreter is a matter of one line.

$ cd my_project
$ maturin develop

This command builds the Rust native module and deploys it in the current virtualenv.

import my_project
  
print(my_project.is_prime(12))
print(my_project.is_prime(11))
  
> False
> True

Behind the hood, the command maturin develop :

• Compile the native Rust module with Cargo: a shared library is compiled and copied in the local python module.
• Install the Python module: the module is installed in your virtualenv. For immediate testing without rebuilding your project on each Python code change, you can use editable installs (i.e. pip install -e) by adding the following lines in your pyproject.yml (Maturin Editable Installs).

[build-system]
requires = ["maturin>=0.12"]
build-backend = "maturin"

Testing your module

We can add a simple property-based test in Python to check the behavior of our is_prime Rust function.

We first add a dependency for the Hypothesis Python package. To add dependencies to our project, we can edit the pyproject.toml file. Maturin supports the PEP 621, enabling Python metadata specification.

→ toml «my_project/pyproject.toml» =
  
[build-system]
requires = ["maturin>=0.12"]
build-backend = "maturin"
  
[project.optional-dependencies]
test = [
  "hypothesis",
  "sympy"
]
  
[project]
name = "my_project"
requires-python = ">=3.6"
classifiers = [
    "Programming Language :: Rust",
    "Programming Language :: Python :: Implementation :: CPython",
]

We can now run the maturin develop command.

$ cd my_project
$ maturin develop --extras test (1)

(1) With the option --extras test Maturin installs Python test dependencies.

We add a simple property-based test.

→ Python «my_project/test/test.py» =
  
from hypothesis import settings, Verbosity, given
from hypothesis import strategies as st
from sympy.ntheory import isprime
import my_project
  
@given(s=st.integers(min_value=1, max_value=2**10))
@settings(verbosity=Verbosity.normal, max_examples=500)
def test_is_prime(s):
    assert isprime(s) == my_project.is_prime(s)
  
if __name__ == "__main__":
       test_is_prime()

We can finally run our property-based Python tests.

$ cd my_project
$ python test/test.py

And our Rust tests.

$ cd my_project
$ cargo test

Wrap-up

That’s it! Your mixed Python/Rust module is ready to be used, deployed, and published. Maturin provides several commands to help compile and distribute your mixed Rust/Python package.

A simple wrap-up :

• There are many ways to improve your Python code. Directly in Python with Numba or in a typed DSL with Cython. You can expose C/C++ native code with binding generators. But you can now benefit from the safety of the Rust language in Python with native performance using PyO3.
• Creating mixed Python + Rust projects is now easy with the Maturin project.
• The final project is packaged as a Python wheel containing Python code and a shared library.

To go further in this topic, I use the previous code and other approaches in this simple benchmark.

A (Simple) Native Python Extension Benchmark

References

[1] “Stack overflow developer survey 2021,” Stack overflow. 2021. https://insights.stackoverflow.com/survey/2021