Welcome Apache Liminal (Incubating): Getting ML apps to production made easy

The challenges involved in operationalizing machine learning models are one of the main reasons why many machine learning projects never make it to production. The process involves automating and orchestrating multiple steps which run on heterogeneous infrastructure — different compute environments, data processing platforms, ML frameworks, notebooks, containers and monitoring tools.

We’ve recently announced the Apache Liminal project incubating under the Apache incubator. Apache Liminal aims to solve the problem of orchestrating and monitoring data and ML systems.

In this post we’re going to introduce the basics of Apache Liminal.

Liminal Basics

Liminal abstracts away from the user various layers of infrastructure such as containers (e.g. docker), runtime (e.g. kubernetes), deployment, monitoring and so forth.

Liminal creates for the user docker images from their code as well as scheduled / constantly running apps which run these images.

Liminal can be easily integrated with an organization’s CI pipelines to allow users to deploy their applications with ease.

One of the guiding principles in Liminal’s design is to intrude as little as possible on data engineers and scientists’ code base. In accordance with this principle, all that is asked of a Liminal user is to add a liminal.yml file to their code repository.

This reduces the amount of different technologies and languages users need to familiarize themselves in order to orchestrate an end-to-end data system.

Defining Liminal Systems

liminal.yml file is composed of a few main sections which help you define your liminal system:

---
name: MyDataScienceApp
owner: Bosco Albert Baracus

First we simply specify the name and owner of the liminal system we are creating.

Services

services:
  - service:
    name: my_datascience_server
    type: python_server
    description: my ds server
    image: myorg/mydatascienceapp
    source: .
    endpoints:
      - endpoint: /predict
        module: serving
        function: predict

In the services section we define the applications of our liminal system that are constantly running. For example, a service of type python_server allows users to serve functions from their code as a constantly running HTTP server responding to incoming requests.

A liminal user need not worry about developing an HTTP based service and how they would build and deploy it, rather, using liminal’s python_server service type would handle it for them.

When the user supplies an image name as well as a source location (relative path in their project from liminal.yml location), liminal builds their source code as a docker image with a functioning server which serves user code. All the user has to do is define the endpoints they want to expose and which functions in their codes should be invoked.

Pipelines

pipelines:
  - pipeline: my_datascience_pipeline
    start_date: 1970–01–01
    timeout_minutes: 45
    schedule: 0 * 1 * *
    metrics:
      namespace: DataScience
    tasks:
      - task: train
        type: python
        description: train model
        image: myorg/mydatascienceapp
        cmd: python -u training.py train
      - task: validate
        type: python
        description: validate model and deploy
        image: myorg/mydatascienceapp
        cmd: python -u training.py validate

In the pipelines section we define pipelines consisting of a set of tasks that are run sequentially and is triggered on a schedule.

Pipelines are useful for purposes such as data ETLs, data science model training and many others.

The list of tasks which comprise a pipeline can enable users to run different types of tasks in their pipeline. For example, using liminal’s python task type a user can easily run parts of their code as a task within a scheduled pipeline of tasks.

A liminal user would not need to know how to develop pipelines but rather receives this functionality from liminal and only needs to define the tasks they wish to run. Similarly to services, a user does not need to create images by themselves but liminal will create it for them. In the example file above we re-use the same image created for the service as both use the same python codebase that belongs to the user.

Running your Liminal system

In order to run liminal system locally we can utilize the liminal cli commands:

liminal build — builds images from user code

liminal deploy — deploys liminal.yml files to liminal server

liminal start — starts liminal server, the current available implementation uses Apache Airflow as a server to run pipelines (as Airflow DAGs)

Below is what the liminal system we create in the above liminal.yml looks like when running:

Running DAG corresponding to user’s liminal.yml on Apache Airflow:

Request to user’s python service:

Python service log:

Try it out yourself

Try it out yourself with Liminal getting started example:
https://github.com/apache/incubator-liminal/blob/master/docs/getting_started.md

What’s next for Liminal

CI Integrations — Simple integrations of liminal with existing organization’s stacks

User Interface — User interface to define liminal systems via wizards instead of yml files

ML Integrations — (Kubeflow, MLflow, Feature store, experiment tracking, model store and more)

Join the effort

Liminal is working under Apache incubator and according to the Apache way. As such, we are very open to contributions and want to grow a robust community to achieve greatness together.

Please join us on our mailing lists at http://liminal.apache.org/

Contribute code and documentation on GitHub at https://github.com/apache/incubator-liminal

Report/Implement issues and features on Apache JIRA