Integrating the Rust toolchain into Rtools (2)

TL;DR

Integrating the Rust toolchain into Rtools is surprisingly straightforward thanks to shared adherence to the standardized C ABI, specifically the Universal C Runtime (UCRT). Although Rust and R may be built using different toolchains, the UCRT ensures binary compatibility, enabling seamless interaction between Rust extensions and the R runtime.

Recently, I attempted to integrate the Rust toolchain into Rtools by simply dragging the binary files for `rustc` and `cargo` from the Rust x86_64-pc-windows-gnu tarball into Rtools45. Surprisingly, it worked perfectly. I described in a previous post.

Integrating the Rust toolchain into Rtools

Why does this work? How can `rustc` and `cargo` operate with R without being compiled using the same toolchain?

The binaries need to interact

When creating a Rust extension for R using {extendr} or {Savvy}, we expose Rust functions as routines callable from R. Sometimes, these Rust functions rely on other libraries that must be linked and called by the Rust binary.

There are three components:

• Binary library (optional): Required by the Rust extension.
• Rust binary: The Rust extension itself.
• R binary: Used to execute R code.

In reality, these three components communicate seamlessly, forming the foundation of writing Rust extension for R.

Binaries are communicated using the C ABI

What’s ABI:

An ABI (Application Binary Interface) specifies how an executable binary can be invoked by other programs. It encompasses two main aspects.

• The layout of data in memory.
• The method of function invocation.

ABI Compatibility

library ABI = API + compiler

To ensure ABI compatibility, the API must first be consistent.

Imagine Rust provides a function with two parameters, but R asks three. This mismatch will cause issues.

Since API compatibility is determined by how the Rust code is written, let’s focus on the compiler.

The second consideration is the compiler.

For instance, with different compiler or compiler flag, function names might change, and the memory layout of data could differ.

One way to ensure compliance with compiler compatibility is by using the same compiler version and compilation flags. By compiling binaries with the same compiler/toolchain, we can achieve ABI compatibility.

Universal standardized C ABI

However, is using the same compiler the only way to ensure ABI compatibility?

The answer is NO.

We can standardize the ABI makes it independent of the compiler.

The UCRT is now a Windows component, and ships as part of Windows 10 and later. The UCRT supports a stable ABI based on C calling conventions, and it conforms closely to the ISO C99 standard, with only a few exceptions. It’s no longer tied to a specific version of the compiler. — Microsoft document

On Windows, there is a C ABI standard known as UCRT (Universal C Runtime). By adhering to UCRT, it ensures consistent memory data layout and function calling conventions.

Only object files compiled for the same C runtime can be linked together on Windows. — Rblog

Since R4.2/Rtools42, it transform to UCRT in windows platform.

What’s more, the Rust gnu toolchains also following the UCRT standard.

By both R and Rust adhering to the UCRT, you don’t need the same compiler to ensure ABI compatibility.

This ensures that the Rust compiler can work with Rtools toolchain.

Portability of the Rust compiler

A small example

We had solve the ABI compatibility issue, now let’s explore the compiling and linking process of Rust extension with a detailed example. It is a toy R package with Rust extension, you can find the code here

Here let’s read the Rust code first.

use extendr_api::prelude::*;
// openssl need c/c++ library
use openssl::ssl::{SslConnector, SslMethod};

#[extendr]
fn check_ssl() -> i32{ // the function will return an integer
    let _ = SslConnector::builder(SslMethod::tls()).unwrap();
    1 + 1
}

// Macro to generate exports
extendr_module! {
    mod test;
    fn check_ssl;
}

The Rust extension interfaces with C/C++ libraries to return an integer to R. It depends on several C/C++ libraries, including ssll, crypto, and zlib, etc. for proper functionality.

As mentioned earlier, C/C++ libraries can easily communicate with Rust using the C ABI under the UCRT runtime, facilitating interaction between Rust and R.

Rust uses Rtools linker and binary library

However, when moving the Rust binary to Rtools, how can we locate binary libraries like ssl, crypto, and zlib?

In order to answer this question, we need step into the Make file.

TARGET = $(subst 64,x86_64,$(subst 32,i686,$(WIN)))-pc-windows-gnu

TARGET_DIR = ./rust/target
LIBDIR = $(TARGET_DIR)/$(TARGET)/release
STATLIB = $(LIBDIR)/libtest.a
PKG_LIBS = -L$(LIBDIR) -ltest -lssl -lcrypto -lz -lws2_32 -lgdi32 -lcrypt32 -lR -lkernel32 -ladvapi32 -lntdll -luserenv -lws2_32 -ldbghelp

all: C_clean

$(SHLIB): $(STATLIB)

CARGOTMP = $(CURDIR)/.cargo

$(STATLIB):
 mkdir -p $(TARGET_DIR)/libgcc_mock
 # `rustc` adds `-lgcc_eh` flags to the compiler, but Rtools' GCC doesn't have
 # `libgcc_eh` due to the compilation settings. So, in order to please the
 # compiler, we need to add empty `libgcc_eh` to the library search paths.
 #
 # For more details, please refer to
 # https://github.com/r-windows/rtools-packages/blob/2407b23f1e0925bbb20a4162c963600105236318/mingw-w64-gcc/PKGBUILD#L313-L316
 touch $(TARGET_DIR)/libgcc_mock/libgcc_eh.a

 # CARGO_LINKER is provided in Makevars.ucrt for R >= 4.2
 if [ "$(NOT_CRAN)" != "true" ]; then \
  export CARGO_HOME=$(CARGOTMP); \
 fi && \
  export CARGO_TARGET_X86_64_PC_WINDOWS_GNU_LINKER="$(CARGO_LINKER)" && \
  export LIBRARY_PATH="$${LIBRARY_PATH};$(CURDIR)/$(TARGET_DIR)/libgcc_mock" && \
  cargo build --target=$(TARGET) --lib --release --manifest-path=./rust/Cargo.toml --target-dir $(TARGET_DIR)
 if [ "$(NOT_CRAN)" != "true" ]; then \
  rm -Rf $(CARGOTMP) && \
  rm -Rf $(LIBDIR)/build; \
 fi

C_clean:
 rm -Rf $(SHLIB) $(STATLIB) $(OBJECTS)

clean:
 rm -Rf $(SHLIB) $(STATLIB) $(OBJECTS) $(TARGET_DIR)

By default, the Rtools linker is utilized by configuring the environment function during the compilation process:

CARGO_TARGET_X86_64_PC_WINDOWS_GNU_LINKER="x86_64-w64-mingw32.static.posix-gcc.exe"

Since the Rtools linker was used, the Rtools C/C++ binary library was defined in the search path.

You can quickly verify the library search path by running the following command in the Rtools toolchain folder. The output will include the lib proved by Rtools.

./x86_64-w64-mingw32.static.posix-gcc.exe -print-libgcc-file-name

Copy Rust binary and its dependencies is enough

Since we are using the linker and libraries from Rtools, we no longer need them from the Rust toolchain. We only require the Rust compiler binaries (rustc, cargo) and their dependencies.

On the Windows platform, the Rust compiler is statically linked, meaning it has minimal dependencies and won’t conflict with the Rtools toolchain.

Although the Rust compiler relies on the MSYS2 shell environment, as explained in the last post, this environment is included with Rtools, so users don’t need to install it separately.

Therefore, by copying the Rust compiler binaries into Rtools folder, they can function perfectly out of box.

Conclusion

Integrating the Rust toolchain into Rtools is surprisingly straightforward thanks to shared adherence to the standardized C ABI, specifically the Universal C Runtime (UCRT). Although Rust and R may be built using different toolchains, the UCRT ensures binary compatibility, enabling seamless interaction between Rust extensions and the R runtime.

This approach offers a portable, low-friction way to develop Rust extensions for R on Windows — no additional installations or patching required.

Appendix: Testing

Lastly, I conducted additional tests on the new customized Rtools (see my last post). All tested packages based on {extendr} and {Savvy} compiled successfully. These packages, including string2path, helloextendr, r-polars, and CellBarcodeRS, also passed their tests and examples. The h3o package compiled and ran examples successfully, but its test suite was unavailable.

Following are the testing code:

library(git2r)

#' # Test Savvy based package

#' ## string2path

#' [Yes] Compile
#' [Yes] Test
#' [Yes] Examples

getwd()
clone("https://github.com/yutannihilation/string2path.git", "string2path")
setwd("string2path")
devtools::load_all()
devtools::document()
devtools::test()
devtools::run_examples()
setwd("..")

#' # Extendr

#' ## helloextendr

#' [Yes] Compile
#' [Yes] Test
#' [Yes] Examples

getwd()
clone("https://github.com/extendr/helloextendr.git", "helloextendr")
setwd("helloextendr")
devtools::load_all()
devtools::document()
devtools::test()
devtools::run_examples()
setwd("..")


#' ## pola-rs/r-polars

#' [Yes] Compile
#' [Yes] Test
#' [Yes] Examples

getwd()
# clone tag v0.22.3
clone("https://github.com/pola-rs/r-polars.git", "r-polars")
setwd("r-polars")
devtools::load_all()
devtools::document()
devtools::test()
devtools::run_examples()
setwd("..")

#' ## h3o

#' [Yes] Compile
#' [NAN] Test
#' [Yes] Examples

getwd()
clone("https://github.com/JosiahParry/h3o.git", "h3o")
setwd("h3o")
devtools::load_all()
devtools::document()
devtools::run_examples()

#' ## CellBarcodeRS

#' [Yes] Compile
#' [Yes] Test
#' [Yes] Examples

getwd()
clone("https://github.com/Wenjie1991/CellBarcodeRS.git", "CellBarcodeRS")
setwd("CellBarcodeRS")
devtools::load_all()
devtools::document()
devtools::run_examples()
devtools::test()
setwd("..")


#' ## test_rust

#' [Yes] Compile

getwd()
clone("https://github.com/Wenjie1991/test_rust_with_rtools.git", "test_rust")
setwd("test_rust")
devtools::load_all()
setwd("..")