Bringing Python and Rust Together in Harmony for pyQuil® 4.0
By Marquess Valdez, Senior Software Engineer
pyQuil has long been the cornerstone for building and running quantum programs on Rigetti quantum processing units (QPUs) through our Quantum Cloud Services (QCS™) platform. It’s an essential client library for us. However, as we’ve advanced the QCS platform, we’ve reached more and more towards Rust for its performance, type system, and emphasis on correctness. In order to support Rigetti’s growing ecosystem of Rust tools and services, much of the functionality in pyQuil has been superseded by our Rust libraries. Fortunately, Rust lends itself well to being used across foreign function interface (FFI). This is another important benefit of Rust for us, as it is an ideal candidate for bridging the gap between our services and users of high level languages like Python, or low level ones like C.
We are still committed to supporting Python and pyQuil, so we have spent the last year retrofitting pyQuil with our modern Rust SDKs. This foundational change to pyQuil brings the benefits of Rust to users in a transparent way, and brings the enhancements needed to compile and run programs on Rigetti’s fourth generation QPUs. You can learn more about the key changes in our “Introducing pyQuil v4” guide. In the remainder of this article, we’ll discuss some of the challenges and breakthroughs we encountered integrating Rust with Python.
Setting the course
Before we continue, let’s clearly state the two primary goals needed to integrate our Rust SDKs with pyQuil:
1. Build Python packages on top of our existing Rust libraries, without compromising the design or idiomatic “Rustiness” of those Rust libraries.
2. Incorporate these packages into pyQuil, while minimizing breaking changes to existing APIs and behaviors.
Building a Python package from a Rust library
We knew that we wanted our Rust libraries to remain pure Rust libraries, devoid of any Python-specific code or types. Conversely, we wanted to ensure that our Python packages adhered to the expectations of Python developers. These goals are conflicting, so it became evident that the most effective way forward was to keep the core logic in our Rust crate and build a separate crate with Rust bindings for a Python package.
We decided that the PyO3 crate would be our framework of choice for building Python packages in Rust. It is widely used and has great documentation. pyo3offers numerous macros that can be used to wrap your Rust code and expose them as Python objects. These macros annotate the definitions of types and functions, which unfortunately limits their usefulness when attempting to build a Python package from types in an external crate.
The typical workaround involves creating newtype wrappers around external types, but this leads to cumbersome boilerplate. For example, a newtype wrapper lacks the convenience of using pyo3 to generate getter and setter properties. Instead, using a newtype wrapper necessitates manual implementation.
This example from quil-rs illustrates the problem. In Quil, an EXCHANGE a b instruction exchanges the values in memory references a and b. This is represented in quil-rs using a MemoryReference and Exchange struct:
pub struct MemoryReference {
pub name: String,
pub index: u64
}
pub struct Exchange {
pub left: MemoryReference,
pub right: MemoryReference
}If we were to directly wrap this struct with PyO3, we would use the pyclass and pyo3 attributes to make Exchangeand MemoryReference Python classes complete with getters and setters for each of their fields:
use pyo3::pyclass;
#[pyclass(get_all, set_all)]
pub struct MemoryReference {
pub name: String,
pub index: u64
}
#[pyclass(get_all, set_all)]
pub struct Exchange {
pub left: MemoryReference,
pub right: MemoryReference
}While convenient, this approach would necessitate injecting Python-specific code and dependencies into our Rust libraries, compromising their purity. But how should we handle code from an external crate?
First, we must create newtype wrappers around external types to apply the #[pyclass] attribute to them:
use quil_rs::instruction::{Exchange, MemoryReference};
use pyo3::prelude::*;
#[pyclass(name = "MemoryReference")]
pub struct PyMemoryReference(MemoryReference);
#[pyclass(name = "Exchange")]
pub struct PyExchange(Exchange)Next, since we can’t use get_all and set_all on our new type wrappers to access the inner fields of MemoryReference and Exchange, we must manually implement getters and setters for each field of the inner type:
#[pymethods]
impl PyMemoryReference {
#[getter]
fn get_name(self) -> String { ... }
#[setter]
fn set_name(self, name: String) -> PyResult<()> { ... }
#[getter]
fn get_index(self) -> u64 { ... }
#[setter]
fn set_index(self, index: u64) -> PyResult<()> { ... }
}
#[pymethods]
impl PyExchange {
#[getter]
fn get_left(self) -> MemoryReference { ... }
#[setter]
fn set_left(self, memory_reference: PyMemoryReference) -> PyMemoryReference { ... }
#[getter]
fn get_right(self) -> MemoryReference { ... }
#[setter]
fn set_right(self, memory_reference: PyMemoryReference) -> PyMemoryReference { ... }
}This approach sacrifices much of the convenience PyO3 offers, is prone to errors, and substantially increases the boilerplate required to maintain a Python package built on an external Rust crate. This was a significant problem for us, especially since quil-rs leans heavily on Rust’s type system to represent Quil programs.
What if we could have the best of both worlds? That’s the goal of rigetti-pyo3, an open-source library we built to extend the capabilities of pyo3by introducing traits and macros that drastically reduce the boilerplate required to wrap external Rust types. With rigetti-pyo3, we can use the py_wrap_data_struct! macro to generate newtype wrappers, complete with getters and setters for each field. All we need to do is specify the fields, the expected Rust types, and the Python-compatible types for conversion:
py_wrap_data_struct! {
PyMemoryReference(MemoryReference) as "MemoryReference" {
name: String => Py<PyString>,
index: u64 => Py<PyInt>
}
}
py_wrap_data_struct! {
PyExchange(Exchange) as "Exchange" {
left: MemoryReference => PyMemoryReference,
right: MemoryReference => PyMemoryReference
}
}rigetti-pyo3 also contains a collection of macros that make it easy to exploit trait implementations on the base type for Python methods. For example impl_hash! uses the Hash implement on the wrapped Rust type to implement Python’s __hash__ method on the wrapped type.
These macros don’t just reduce boilerplate, they also make our Python API more consistent by ensuring every binding implements common functionality in the same way. A good example of this is the py_wrap_union_enum! macro, which wraps a tagged union (or a Rust enum with variants) with a straightforward API for constructing and interacting with Rust enums as Python classes.
rigetti-pyo3 has proved to be an invaluable framework for building Python packages on top of an external Rust crate. It has enabled us to build a seamless integration between our Rust libraries and a Python counterpart, without making sacrifices in the design of either.
Retrofitting pyQuil
While pyQuil and our Rust libraries solve common problems, their approaches to the solutions are often quite different. In many cases, the only similarity in their approaches is in their inflexibility. pyQuil users have certain expectations about how pyQuil should work that we didn’t want to break. Similarly, we didn’t want to compromise on the design of our Rust libraries to satisfy pyQuil’s existing API. We had to connect the dots in a way that didn’t sacrifice the quality of either library. This posed some unique challenges.
For one, there is a lot of overlap, but each also offers a lot of functionality the other doesn’t. Generally, adding new features from our Rust libraries into pyQuil wasn’t a challenge, as we had flexibility in how to incorporate them. However, in the cases where pyQuil had more functionality, we often had to backport it back to our Rust libraries. We had to be vigilant in our decision-making here. We wanted to backport anything necessary to provide a complete and coherent API, but at the same time, we didn’t want to overcommit and port pyQuil-specific functionality back to our Rust SDKs.
Another challenge was in how to meet the expectations of pyQuil’s existing API without compromising the API provided by our Rust SDKs. One example of this involves asycnio and pyQuil’s lack of support for it.
The Async Conundrum
Much of our Rust API interacts with external services over a network, these task naturally lend themselves to async Rust. While pyo3alone doesn’t directly support async functions, the fantastic pyo3-asyncio makes it trivial to expose async Rust functions as Python asyncio functions. However, pyQuil doesn’t use asyncio in its own API and using these asyncio functions as-is would require that pyQuil introduce the async keyword on many of its core methods. This would, in turn, require users to adopt asyncio as well. This would be a major breaking change for users, one we weren’t willing to make.
At first, we tried to wrap the asyncio functions exported by our Rust bindings in Python by manually calling the asyncio event loop API to run the future in a synchronous function. This path didn’t get us very far, and all the variations on the idea were dubious at best. In the end, none performed acceptably in both synchronous and asynchronous contexts.
Instead, what if we push all the asynchronous machinery into the Rust runtime? This comes with its own set of challenges. For one, we wanted to make sure that we were handling OS signals appropriately. It’s not uncommon for users to want to bail on a long running function by hitting Ctrl-C, which sends a SIGINT signal to the running program. In the case of Python programs, the running Python interpreter needs to handle these signals, which means while Rust has control, signals won’t be processed. This gotcha is documented by pyo3 and is something we needed to be aware of when attempting to make potentially long running asynchronous functions synchronous. One further complication in all of this is that the Python API function called to check the signals from Rust, `PyErr_CheckSignals()`, must be called on the main thread, otherwise the call is a no-op.
In summary, we needed to wrap an async Rust function such that it appeared synchronous in Python, while also making sure to handle signals on the main thread, so OS signals are respected.
Let’s do it. Given a contrived async Rust function foo:
async fn foo() -> String {
tokio::time::sleep(Duration::from_secs(3));
"hello".to_string()
}With pyo3_asyncio we can export this as an asyncio function:
#[pyfunction]
fn py_foo_async(py: Python<'_>) -> PyResult<&PyAny> {
pyo3_asyncio::tokio::future_into_py(py, async { Ok(foo().await) })
}But how can we wrap it in a synchronous API? First, we get the current runtime, then we spawn our async function as a task on that runtime. Then we can use tokio::select! to manage returning the result from our task, or from the signal handler, whichever returns first. Wrap all of that in the current runtime, and — voila! — we have a synchronous Python function that’s using Rust’s own async runtime behind the scenes:
#[pyfunction]
fn py_foo_sync() -> PyResult<String> {
let runtime = pyo3_asyncio::tokio::get_runtime();
let handle = runtime.spawn(foo());
runtime.block_on(async {
tokio::select! {
result = handle => result.map_err(|err| pyo3::exceptions::PyRuntimeError::new_err(err.to_string())),
signal_err = async {
let delay = std::time::Duration::from_millis(100);
loop {
Python::with_gil(|py| {
py.check_signals()
})?;
tokio::time::sleep(delay).await;
}
} => signal_err
}
})
}This is great, but it’s a lot to do for every async function. Instead of repeating this setup for each async function in our API, we can — you guessed it! — use a macro.
macro_rules! py_sync {
($py: ident, $body: expr) => {{
$py.allow_threads(|| {
let runtime = ::pyo3_asyncio::tokio::get_runtime();
let handle = runtime.spawn($body);
runtime.block_on(async {
tokio::select! {
result = handle => result.map_err(|err| ::pyo3::exceptions::PyRuntimeError::new_err(err.to_string()))?,
signal_err = async {
let delay = ::std::time::Duration::from_millis(100);
loop {
::pyo3::Python::with_gil(|py| {py.check_signals()})?;
::tokio::time::sleep(delay).await;
}
} => signal_err,
}
})
})
}};
}One addition to our macro is how we’ve wrapped everything in py.allow_threads. This releases the global interpreter lock (GIL) so that other Python threads can run while we are doing Rust-only work. We only re-acquire the GIL when we need to check for OS signals using Python::with_gil.
Now for any async function, we can just write:
#[pyfunction]
fn py_foo(py: Python<'_>) -> PyResult<String> {
py_sync!(py, async { Ok(foo().await) })
}This is great too, but we can take it even further. These synchronous functions are great for compatibility, but some users might appreciate a proper asyncio API. That’s why we built yet another macro that builds on the last one to provide a synchronous and asynchronous variant of a single async function. This lets us write the function once in its natural async form and get both sync and async variants for free.
// This results in two Python functions:
// def foo(): ...
// async def foo(): ...
py_sync::py_function_sync_async! {
#[pyfunction]
async fn foo() -> PyResult<String> {
Ok(foo().await)
}
}Being able to continue to support a synchronous API while not missing out on the opportunity to provide an asynchronous one is a huge win for us, and a good example of how combining Rust with Python can bring benefits not easily achieved with Python alone.
The Payoffs: Functionality and Performance
We’ve established that there are challenges in bridging the gap between existing Python and Rust libraries in a way that doesn’t compromise on the quality or user experience of either. So what did it get us?
As mentioned earlier, our Rust libraries have begun to supersede pyQuil in functionality. Most importantly, they bring the enhancements needed to compile and run programs on Rigetti’s next-generation Ankaa systems.
Furthermore, by consolidating the logic behind parsing and serializing Quil programs, building them programmatically, and executing and retrieving job results into our Rust libraries, we’ve built a solid foundation for pyQuil now and in the future. Using the same logic in our services and client libraries makes it easier for us to maintain and extend pyQuil while also providing a more consistent experience for users.
Finally, we can’t end a Python and Rust blog post without mentioning performance. By porting core logic to Rust, we’ve seen significant performance gains in many areas, such as parsing and serializing Quil programs. This is essential, since parsing and serialization are key steps in the compile-and-execute workflow that is so common in pyQuil.
Conclusion
Bringing Python and Rust together for pyQuil v4 presented many challenges. From the initial decision to build on top of our existing Rust libraries without compromising their design, to meeting the expectations of longtime pyQuil users without introducing breaking changes, we navigated a complex path. Through these efforts, we’ve modernized pyQuil, providing users with the performance and type safety benefits of Rust while maintaining the familiarity and ease of use of Python.
If you are interested in learning more about quantum programming with pyQuil, check out the pyQuil documentation.
