Introducing VlConvert
Embedding JavaScript in Python with Rust
Background
As I’ve written about previously, VegaFusion is a project that provides server-side scaling for data visualizations specified using the Vega visualization grammar. The core VegaFusion runtime is written is Rust and embedded as a native Python library.
Vega-Lite is a higher-level visualization grammar that compiles into Vega. The popular Altair library provides a Pythonic API for creating Vega-Lite visualizations, where the actual compilation of Vega-Lite to Vega is implemented by the Vega-Lite Typescript library.
Currently, VegaFusion’s Python support is tightly coupled to a Jupyter Widget which performs this Vega-Lite to Vega compilation in the web-browser. I would like to break this tight coupling by moving the Vega-Lite to Vega compilation to Python, making it possible to use VegaFusion in Python environments that support Vega but not Jupyter Widgets.
The Challenge
This raises the question, how can a Typescript library be embedded in Python? Vega and Vega-Lite support running in Node.js, so one option is to have the Python library call out to Node.js as a subprocess. This works, but it’s not very satisfying as it requires the end user to install Node.js, which is not available on PyPI and so cannot be installed automatically as a dependency when installing VegaFusion. It also requires somehow distributing a JavaScript bundle that includes Vega-Lite and all of its dependencies.
What I really want is a way to embed the Vega-Lite bundle and the Node.js runtime (or something like it) directly into a self-contained native Python library.
Potential Embedding Solutions
Node.js in C++
Based on the documentation, it is possible to embed Node.js into a C++ application, and so it should be possible to wrap the Node.js runtime in Python by adding a C-shim around a custom C++ library. Or perhaps Boost Python could be used to interface with Node.js directly. Regardless, I don’t enjoy working in C++ (or dealing with C++ build systems), so I didn’t seriously consider this option.
Deno in Rust
Deno is an exciting project created by Ryan Dahl (The original author of Node.js). As described on the Deno website:
Deno is a simple, modern and secure runtime for JavaScript, TypeScript, and WebAssembly that uses V8 and is built in Rust.
If you’re interested in learning more about Deno, I encourage you to check out the project website and this excellent blog post by LogRocket.
For the problem at hand, the key feature of Deno is that it’s developed as a collection of Rust crates that can be easily embedded in a Rust project. Such a project can then be exposed to Python using the excellent PyO3 crate.
This approach, wrapping a core Rust library for use in Python with PyO3, is exactly how VegaFusion works, and so it was an appealing starting point for this project.
VlConvert
I’ve created a new project called VlConvert (GitHub, PyPI) that performs Vega-Lite to Vega compilation using the Deno in Rust approach described above.
This particular library is pretty niche, but I wanted to share some details on how it works in case they are useful to others who might want to accomplish something similar. All in all, I found this to be a pretty elegant approach to making a JavaScript library available in Python.
Vendoring Vega-Lite
The vl-convert-vendor crate in the VlConvert project is responsible for downloading the Vega-Lite source code (and all of its dependencies) using deno vendor and creating the import_map.rs source file that embeds the contents of all of the vendored JavaScript files as Rust strings using the include_str! macro.
One thing to note, Vega-Lite and it’s dependencies are pulled from the Skypack CDN, which is designed to work well with Deno. See the Packages from CDNs section of the Deno documentation for more info on what that means.
Rust library
The vl-convert-rs crate (GitHub, crates.io) is a pure Rust library that interfaces with Deno. It implements a custom module loader (as an alternative to Deno’s built-in FsModuleLoader used in the deno_core examples) that loads module source code from import_map.rs. This is how VlConvert is able to embed the Vega-Lite source code directly in the library binary.
After initializing Vega-Lite, conversion requests are handled by executing JavaScript snippets that return the converted Vega specifications as strings.
CLI application
The vl-convert crate (GitHub, crates.io) uses clap to wrap the vl-convert-rs library as a simple CLI application. The vl-convert CLI can be installed using cargo with:
$ cargo install vl-convert
$ vl-convert --helpCLI application for converting Vega-Lite visualization specifications to Vega specifications
Usage: vl-convert [OPTIONS] --input-vegalite-file <INPUT_VEGALITE_FILE> --output-vega-file <OUTPUT_VEGA_FILE>
Options:
-i, --input-vegalite-file <INPUT_VEGALITE_FILE>
Path to input Vega-Lite file
-o, --output-vega-file <OUTPUT_VEGA_FILE>
Path to output Vega file to be created
-v, --vl-version <VL_VERSION>
Vega-Lite Version. One of 4.17, 5.0, 5.1, 5.2, 5.3, 5.4, 5.5 [default: 5.5]
-p, --pretty
Pretty-print JSON in output file
-h, --help
Print help information
-V, --version
Print version information
If there is demand, this CLI application could also be published as a self-contained binary for each operating system.
Python library
The vl-convert-python crate (GitHub, PyPI) uses PyO3 to wrap vl-convert-rs as a Python library. Using the maturin build tool, this crate is compiled to native Python wheels for Linux, MacOS, and Windows.
A result that I’m very happy with is that the compiled wheels are only ~14MB in size. While not apples-to-apples, this compares quite favorably to the ~30MB size of the compressed Node.js binary for MacOS.
Future work: Image export
A natural extension of VlConvert is to support static image export. Vega already supports static image export in Node.js, so this almost already works. The difficulty is that Vega, when running in Node.js, relies on the node canvas library for text measurements and raster image generation. Unfortunately, the node canvas library does not work in Deno.
An approach I would like to explore in the future is to update Vega to support custom text measurement calculations, and then try using resvg for text measurement and svg to png rasterization.
Update — 10/17/2022
The image export approach overviewed above worked out well! It turned out to be straightforward to override Vega’s text measurement functionality, so no change was required in Vega itself to achieve accurate text placement using resvg instead of node canvas. Version 0.3.0 of VlConvert (both the Python library and CLI application) now supports both SVG and PNG static image export.
Try it out now on Binder!
