Connecting Python and Java with Rust
Which one to choose — Java? Python? Most organizations use both, no need to say more here. Two of the top used languages around.
Sometimes we need to share logic between the two. How? Writing the logic twice in each language when the logic becomes more complex and constantly changes is a big no no. So what can we do?
Sharing Logic Between Java & Python
Option 1 - Micro Service
Create a (micro) service written in one of the two languages and share an API (e.g. rest / gRPC) for the shared logic.
A service has drawbacks:
- Communication — Sometimes the two systems (Python & Java) are disconnected from each other, like a Java mobile app and Python backend ML pipeline.
- SLA — Sometimes one team owns and updates the logic. For example a team of data scientists owns an algorithm in Python to be shared with a production system written in Java. They probably lack the skills/resources to maintain a micro-service with high-availability requirements to an engineering team running production system using Java.
- Performance — calling a service has performance limitations. Even with today’s technologies (like gRPC) adding 50ms to each request might be too much for some realtime systems.
Option 2 - Shared Library
Writing a shared library for both, using Python or Java as the language for the shared library is tricky. Some of the issues — two VMs, two GCs, multithreading, serialization, performance penalty… 😕
There are some projects to import Java from Python (which seem less production ready). Taking a look at how PySpark solved it gives us a glance on the complexity of this issue.
Rust to the Rescue
Rust comes to the rescue — as the shared library with bindings for both languages (link to the full project on github).
Many popular languages support Native Extensions/FFI including Java & Python. Because Rust has no GC and can be compiled to shared library like C does (.SO /.DLL file) there are crates for most popular languages for writing native extensions in Rust, we’ll use the following crates today:
We’ll use Cargo workspace feature to manage multiple packages.
Lets get Started!
project files overview:
Step 1 — write the logic in Rust library
Lets create our package that contains our shared logic - rsdivider package. In this example we’ll just share a div_numbers function:
add the package’s Cargo.toml file:
nothing fancy so far.
Step 2 — wrap the logic for Python
Now lets use PyO3 to wrap the rsdivider package as Python module:
The code is straightforward here, we create a python function and add it to our module. On line 3 we import our shared library written on step 1.
To compile it we can run cargo build --release from the root directory.
our python module can be used like this:
>>> import rsdivider_py
>>> rsdivider_py.div_numbers(4,2)
we create a library named rsdivider_py which is C dynamic library (a C compatible library).
we’ll use the shared library as a dependency to the path of rsdivider to always build and use the latest source.
note: for PyO3 we need the specialization feature of rust (i.e. nightly rust version). you can follow this issue for resolution of when PyO3 supports stable Rust. There is alternative crate rust-cpython that offers a similar way to wrap python and can be built with stable rust version.
Step 3— wrap the logic for Java
Similar to Python’s wrapper, lets create another package called rsdividerjava
pub extern tells the compiler to export the function in our lib in a C style way (i.e. can be used like it was compiled from C).
#[no_mangle] - keep Rust from mangling the name of our function.
#[allow(non_snake_case)] - Turn off linter warnings about naming conventions, we need to use Java_{lib}_{function} as the function name so it can be imported nicely from Java.
jdouble - We convert from java-double into float64 and returns the result as jdouble again.
Our Cargo.toml, similar to Python’s:
Example usage from java:
Before this will work, we need to create .h file for our lib:
$ javah RsDivider
we will get RsDivider.h file on same directory that looks like this:
Now we got a native extension used from our Java main.
Summary
We can share logic between Java, Python and many more languages that supports native extensions using rust.
Take a look at this github repo for a full example on how to make it work.
