Integrating Micronaut and Jupyter for Data Science

Dan Stieglitz
Feb 12 · 4 min read
Image for post
Image for post
Photo by NICHOLAS CAPPELLO on Unsplash

Many modern distributed architectures are built on the concept of microservices; small, simple (well, they are supposed to be simple) services that do well-defined tasks and are easily deployable in a highly available infrastructure. A great use for microservices is defining a business domain, and providing a data gateway for applications through a well-defined API. Using microservices, you can deploy this API across a number of physical or virtual machines that scale to meet a large input or output demand.

While microservices provide a sound structural interface to underlying systems and data, they do little to ease the complexity of integrating these systems (and data) into a data science pipeline. Data scientists rely on rigid environments like Jupyter and R to load and process data and produce results. Data is loaded through plugins, scraped from the web, Excel spreadsheets, and other means; loaded into frames and tables, processed and then something, hopefully, is done with the results.

At Stainless AI we wanted to develop a pipeline that could combine the worlds of microservices and data science (specifically Jupyter, with all due respect to our friends who use R), and build an environment where you could load data directly from existing microservices, execute robust data science pipelines, and then use the results immediately via the same set of microservices to transfer data back to the business domain, all within a single session. So we developed the micronaut-jupyter plugin (http://github.com/stainlessai/micronaut-jupyter).

Micronaut (https://micronaut.io) is “A modern, JVM-based, full-stack framework for building modular, easily testable microservice and serverless applications.” Brought to us by our friends at Object Computing, Inc. (https://objectcomputing.com) or OCI — the folks who developed Groovy, Grails and other JVM-based goodies — Micronaut is a fast, memory-efficient platform for building cloud native, serverless or container-driven microservices. Using Micronaut we can develop modern architectures leveraging cloud-native paradigms like serverless to make the most out of our tech spend, but how can we also leverage cutting edge data science tools to maximize the business value of our data?

The Jupyter-Micronaut Plugin

Jupyter (https://jupyter.org) and related projects are are suite of free, open source, web-based tools for interactive programming. Jupyter is very popular in the data science community due to its great user interface (Jupyterlab), and out-of-the-box integration with the Python ecosystem. (R folks, you can use R with Jupyter too!) Jupyter by itself doesn’t do much to help with data engineering tasks like loading and preprocessing data, although many tools exist to do that. Jupyter does provide a flexible kernel system where you can run different languages through their interactive interface.

Jupyter defaults to Python, but if you’re like us and like to leverage the amazing syntactic sugar provided by Groovy for your data science, you’ll want a JVM kernel. The folks at twosigma released BeakerX (http://beakerx.com), which provides this functionality. Using the jupyter-micronaut plugin developed by Stainless AI, we can extend Jupyter to do things like:

Examples

Several example Micronaut applications are provided in the repository. To run them, cd to the example directory and run the following commands:

~/micronaut-jupyter$ cd examples/basic-service~/micronaut-jupyter/examples/basic-service$ ../../gradlew run

The gradle script provided with the examples will run a Micronaut application configured with the plugin, which will install a Micronaut kernel specification into your local Jupyter installation. The kernel runs as a server which will continue to run as long as the gradle process is active.

Now, you can launch Jupyter and you should get a Micronaut kernel option from the launch screen in lab, or through the menu system in the traditional Jupyter interface:

Image for post
Image for post

After launching the Micronaut kernel, you can load the example notebooks and test out the various Micronaut features from your Jupyter installation:

Image for post
Image for post

The query power of GORM, syntactic sugar of Groovy and flexibility of Micronaut brings a new dimension to our data science capabilities by allowing us build advanced architectures like domain-specific languages (DSLs) for our business data. Reach out to us at info@stainless.ai if you’d like to learn how these tools can make a huge impact on your business insights!

Links

Below are all the links referenced in this article:

https://objectcomputing.com

https://jupyter.org

The Signal

Practical notes on artificial intelligence, machine learning and computer vision from Stainless AI

Dan Stieglitz

Written by

Dan is the CEO of Stainless AI, Inc., which provides cognitive computing solutions to businesses through machine learning and artificial intelligence.

The Signal

Technical articles and musings from Stainless AI, a provider of cognitive computing solutions: machine learning, computer vision, data science, and more!

Dan Stieglitz

Written by

Dan is the CEO of Stainless AI, Inc., which provides cognitive computing solutions to businesses through machine learning and artificial intelligence.

The Signal

Technical articles and musings from Stainless AI, a provider of cognitive computing solutions: machine learning, computer vision, data science, and more!

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store