Integrating Micronaut and Jupyter for Data Science

Dan Stieglitz
Feb 12, 2021 · 4 min read
Photo by NICHOLAS CAPPELLO on Unsplash

Many modern distributed architectures are built on the concept of microservices; small, simple (well, they are supposed to be simple) services that do well-defined tasks and are easily deployable in a highly available infrastructure. A great use for microservices is defining a business domain, and providing a data gateway for applications through a well-defined API. Using microservices, you can deploy this API across a number of physical or virtual machines that scale to meet a large input or output demand.

While microservices provide a sound structural interface to underlying systems and data, they do little to ease the complexity of integrating these systems (and data) into a data science pipeline. Data scientists rely on rigid environments like Jupyter and R to load and process data and produce results. Data is loaded through plugins, scraped from the web, Excel spreadsheets, and other means; loaded into frames and tables, processed and then something, hopefully, is done with the results.

At Stainless AI we wanted to develop a pipeline that could combine the worlds of microservices and data science (specifically Jupyter, with all due respect to our friends who use R), and build an environment where you could load data directly from existing microservices, execute robust data science pipelines, and then use the results immediately via the same set of microservices to transfer data back to the business domain, all within a single session. So we developed the micronaut-jupyter plugin (http://github.com/stainlessai/micronaut-jupyter).

Micronaut (https://micronaut.io) is “A modern, JVM-based, full-stack framework for building modular, easily testable microservice and serverless applications.” Brought to us by our friends at Object Computing, Inc. (https://objectcomputing.com) or OCI — the folks who developed Groovy, Grails and other JVM-based goodies — Micronaut is a fast, memory-efficient platform for building cloud native, serverless or container-driven microservices. Using Micronaut we can develop modern architectures leveraging cloud-native paradigms like serverless to make the most out of our tech spend, but how can we also leverage cutting edge data science tools to maximize the business value of our data?

The Jupyter-Micronaut Plugin

Jupyter defaults to Python, but if you’re like us and like to leverage the amazing syntactic sugar provided by Groovy for your data science, you’ll want a JVM kernel. The folks at twosigma released BeakerX (http://beakerx.com), which provides this functionality. Using the jupyter-micronaut plugin developed by Stainless AI, we can extend Jupyter to do things like:

  • Execute Groovy code
  • Import classes on your Micronaut app’s classpath
  • Access Micronaut beans
  • Use Micronaut Data repositories
  • Use GORM Data Services and dynamic finders

Examples

Several example Micronaut applications are provided in the repository. To run them, cd to the example directory and run the following commands:

~/micronaut-jupyter$ cd examples/basic-service~/micronaut-jupyter/examples/basic-service$ ../../gradlew run

The gradle script provided with the examples will run a Micronaut application configured with the plugin, which will install a Micronaut kernel specification into your local Jupyter installation. The kernel runs as a server which will continue to run as long as the gradle process is active.

Now, you can launch Jupyter and you should get a Micronaut kernel option from the launch screen in lab, or through the menu system in the traditional Jupyter interface:

After launching the Micronaut kernel, you can load the example notebooks and test out the various Micronaut features from your Jupyter installation:

The query power of GORM, syntactic sugar of Groovy and flexibility of Micronaut brings a new dimension to our data science capabilities by allowing us build advanced architectures like domain-specific languages (DSLs) for our business data. Reach out to us at info@stainless.ai if you’d like to learn how these tools can make a huge impact on your business insights!

Links

https://objectcomputing.com

https://jupyter.org

The Signal

Practical notes on artificial intelligence, machine learning and computer vision from Stainless AI

The Signal

Technical articles and musings from Stainless AI, a provider of cognitive computing solutions: machine learning, computer vision, data science, and more!

Dan Stieglitz

Written by

Dan is the CEO of Stainless AI, Inc., which provides cognitive computing solutions to businesses through machine learning and artificial intelligence.

The Signal

Technical articles and musings from Stainless AI, a provider of cognitive computing solutions: machine learning, computer vision, data science, and more!