Monitoring ML systems Using MLOps — an Overview

by Venkata Bhargavi S & KrishnaKanth J

What is MLOps (Machine Learning Operations)

A framework aiming at deploying and monitoring machine learning models in production efficiently. It is essentially a combination of Machine Learning, Software Development, and Operations.

The complete MLOps process includes the following list of concepts:

1. Data Engineering
2. Version control of Data, ML models and Code
3. Continuous Integration and Continuous Deployments pipelines,
4. Automating deployments and experiments
5. Model Performance assessment and
6. Model Monitoring in production.

MLOps in Model Training:

MLOps can be used in versioning of code Data can be stored as reference which can help us in identifying the drift in the new data after deployment Metadata of model can be stored
Eg: You trained a dataset with various models with different hyper- parameters, then MLOps log will help in tracking all the experimented models and choose the best one

Storing MetaData

In code, data and model management metadata like pipeline run ID, performed steps, start/end timestamps, train/test dataset split, hyper-parameters, model object, various statistics/profiling, etc.

Versioning of code:

Software development requires the use of code versioning tools which makes it possible for more than one developer to work on the same project simultaneously. Versioning these bases will allow them to work in an organized manner. It is also important to understand reproducibility at the same time. Documenting and reusing important artifacts in a project is essential for making the teams’s project a working model in production.

Model Deployment:

1. Model Registry: Model registry will help us in cataloging production models, version management, managing metadata associated with models, model deployment to production and automating them through CI/CD
2. Model Serving: Model serving refers to making Machine Learning models accessible through API and hosting them on cloud by retrieving the respective model from model registry

In this article we mainly focus on Monitoring using MLOps. Before going into the discussion let’s try to understand why Monitoring is important.

Monitoring ML System using MLOps:

Most models do not predict well as soon as they are deployed into production. It is usually difficult to figure out why the model performance dropped. It can be due to many reasons like the data exposed to the model will be different from what it has been trained on or due to drift in Data or due to drift in relationship between the features and target. To prevent model performance degradation, one needs to identify and address the drift sources in ML systems.

There are majorly 3 types of monitoring:

1. System Monitoring
2. Data Monitoring
3. Model Monitoring

1. System Monitoring:

We must first make sure that model is performing well by checking the actual system whether it is working as expected. System health can be analyzed through some metrics like throughput, error rate, latency, memory utilization, GPU/CPU usage etc.

2. Data Monitoring:

Monitoring just system health won’t be sufficient as we also need to identify the underlying issues with our model. In order to understand the decay of model performance, we must first understand the different types of Drifts that can contribute to it. Monitoring the drift can prevent the model decay by taking required actions on time. Let’s understand each type of drift in detail below.

Drift:

Drift is a change of entity with respect to reference (training data) entity. Production data distribution can change from our baseline data due to several external factors. As a result of changing environments, different drifts lead to reduced prediction power and decay in model performance.

There are three main types of drift listed below.

1. Data Drift
2. Target Drift
3. Concept Drift

a. Data Drift:

which is also known as Feature Drift, in which properties of individual variables (input features) change or change in distribution of input features. There are various causes of data drift including seasonality, unexpected environmental changes etc.

Sample to show the Data Drift.

b. Target Drift: Shift in distribution of Target due to adding or removal of classes in Categorical tasks. Retraining the model can mitigate the performance issues but identifying the actual issue is the critical task in the real world.

c. Concept Drift: Besides the drift in input and output, we can also identify the drift in actual relationship between input features and target.

Consider an example, a model trained on predicting heart disease. Do you think the model trained before Covid would work at par post Covid-19? After the pandemic, health data patterns have changed dramatically and post covid symptoms make it difficult to predict the cause of heart disease. This problem of concept drift occurs when the model based on an old set of data is no longer consistent with the new set of data.

3. Model Monitoring:

Monitoring Machine Learning models in production is a practice used to identify potential issues that can add negative business value. This can help identify issues about prediction quality, data relevance, model accuracy and bias. Monitoring is important because

• It Can track live model performance through evaluation metrics like accuracy, precision, f1 scores etc.
• We can remedy the problem of poor generalization caused by models that were trained on smaller dataset
• Monitoring ML models can assist in understanding and debugging production models thereby reducing complications that arise

Methodology to Detect Drift:

Below table represents each algorithm that works on different kinds of data.

Source: SeldonIO

Top MLOps frameworks for Production Model Monitoring

Evidently:

1. Open-source ML model monitoring system, which generates interactive reports from pandas DataFrame.
2. Ability to capture Data Drift, Target Drift (Numerical and Categorical targets)
3. Ease to integrate, platform-agnostic
4. No Alert Mechanism available
5. Can Log Metrics to MLFlow

Whylogs:

1. Open source, and high-performance statistical data logging library
2. Ability to capture model metrics and Monitor data quality, Collects all possible
3. statistical metrics about structured or unstructured data
4. Ease to integrate, platform-agnostic
5. No Alert Mechanism available

Fiddler:

1. Managed Model monitoring tool has a very simple, clean, interactive dashboard.
2. Visual tool to monitor data drift, model metrics, data integrity and outliers.
3. Alert Mechanism available, lets you set up alerts about production concerns.
4. Overall, it’s a fantastic tool for monitoring machine learning models, with all of the capabilities you’ll need.

Alibi Detect:

1. Open-source python library focused on outlier and drift detection,
2. Ability to capture drift for tabular data, text, images and time-series data
3. Ease to Integrate
4. Log metrics to MLFlow
5. No Alert Mechanism

Conclusion:

To conclude, it is advised to try above frameworks and pick the one best fits for your use case. We hope our analysis helps you find a way to adapt the technology.

If you are interested in trying MLOps framework using Machine Learning model, feel free to use our notebook to predict Heart Disease available in GitHub repository.

You can check out our Heart Disease prediction App to know you Heart Condition.

References:

MLOps Platforms Compared

ml-ops.org

Best End-to-End MLOps Platforms: Leading Machine Learning Platforms That Every Data Scientist Need…

Hope you find this useful!

Follow us on LinkedIn Venkata Bhargavi Sikhakolli and Krishnakanth J