Using Trained Machine Learning Models for Inference in Java

Photo by Akram Huseyn on Unsplash

In this article, we will discuss how to use trained ML models (trained using Pytorch, Tensorflow or any other framework) for inference in Java applications.

Inference is the process of using a trained model to make predictions or decisions on new, unseen data

Let’s Start with the Why?

Why would you want to do something like this?

To understand that, let’s take a look at the available options —

Other than LLMs (which can be accessed using external API calls or specific Java based inference libraries), we don’t have a lot of straightforward options to use other models directly in Java applications.

Models as we know today are mostly trained using Pytorch and Tensorflow. There are hundreds of these models available for free on platforms like HuggingFace.

These are thus easy to integrate in Python environments.

To use them in Java environment, we would either need to deploy it on a server and expose an API that can be consumed, or, convert it to an intermediate format like torchscript/ONNX/GGUF(for LLMs) etc. and use a Java based library that supports that format to directly integrate it…