Feb 12, 2021
Integrating Micronaut and Jupyter for Data Science
Many modern distributed architectures are built on the concept of microservices; small, simple (well, they are supposed to be simple) services that do well-defined tasks and are easily deployable in a highly available infrastructure. A great use for microservices is defining a business domain, and providing a data gateway for applications through a well-defined API. Using microservices, you can deploy this API across a number of physical or virtual machines that scale to meet a large input or output demand.
While microservices provide a sound structural interface to underlying systems and data, they do little to ease the complexity of integrating these systems (and data) into a data science pipeline. Data scientists rely on rigid environments like Jupyter and R to load and process data and produce results. Data is loaded through plugins, scraped from the web, Excel spreadsheets, and other means; loaded into frames and tables, processed and then something, hopefully, is done with the results.
At Stainless AI we wanted to develop a pipeline that could combine the worlds of microservices and data science (specifically Jupyter, with all due respect to our friends who use R), and build an environment where you could load data directly from existing microservices, execute robust data science pipelines, and then use the results immediately via the same set of microservices to transfer data back to the business domain, all within a single session. So we developed the micronaut-jupyter plugin (http://github.com/stainlessai/micronaut-jupyter).
Micronaut (https://micronaut.io) is “A modern, JVM-based, full-stack framework for building modular, easily testable microservice and serverless applications.” Brought to us by our friends at Object Computing, Inc. (https://objectcomputing.com) or OCI — the folks who developed Groovy, Grails and other JVM-based goodies — Micronaut is a fast, memory-efficient platform for building cloud native, serverless or container-driven microservices. Using Micronaut we can develop modern architectures leveraging cloud-native paradigms like serverless to make the most out of our tech spend, but how can we also leverage cutting edge data science tools to maximize the business value of our data?
The Jupyter-Micronaut Plugin
Jupyter (https://jupyter.org) and related projects are are suite of free, open source, web-based tools for interactive programming. Jupyter is very popular in the data science community due to its great user interface (Jupyterlab), and out-of-the-box integration with the Python ecosystem. (R folks, you can use R with Jupyter too!) Jupyter by itself doesn’t do much to help with data engineering tasks like loading and preprocessing data, although many tools exist to do that. Jupyter does provide a flexible kernel system where you can run different languages through their interactive interface.
Jupyter defaults to Python, but if you’re like us and like to leverage the amazing syntactic sugar provided by Groovy for your data science, you’ll want a JVM kernel. The folks at twosigma released BeakerX (http://beakerx.com), which provides this functionality. Using the jupyter-micronaut plugin developed by Stainless AI, we can extend Jupyter to do things like:
- Execute Groovy code
- Import classes on your Micronaut app’s classpath
- Access Micronaut beans
- Use Micronaut Data repositories
- Use GORM Data Services and dynamic finders
Examples
Several example Micronaut applications are provided in the repository. To run them, cd to the example directory and run the following commands:
~/micronaut-jupyter$ cd examples/basic-service~/micronaut-jupyter/examples/basic-service$ ../../gradlew run
The gradle script provided with the examples will run a Micronaut application configured with the plugin, which will install a Micronaut kernel specification into your local Jupyter installation. The kernel runs as a server which will continue to run as long as the gradle process is active.
Now, you can launch Jupyter and you should get a Micronaut kernel option from the launch screen in lab, or through the menu system in the traditional Jupyter interface:
After launching the Micronaut kernel, you can load the example notebooks and test out the various Micronaut features from your Jupyter installation:
The query power of GORM, syntactic sugar of Groovy and flexibility of Micronaut brings a new dimension to our data science capabilities by allowing us build advanced architectures like domain-specific languages (DSLs) for our business data. Reach out to us at info@stainless.ai if you’d like to learn how these tools can make a huge impact on your business insights!
Links
Below are all the links referenced in this article:
