Challenges in operationalizing a machine learning system

Shabaz Patel
Oct 25, 2020 · 5 min read

The goal of this blog is to cover the key topics to consider in operationalizing machine learning and to provide a practical guide for navigating the modern tools available along the way. To that end, the subsequent blogs will include further detailed architecture concepts and help you apply them to your own model pipelines.

This blog series will not explain machine learning concepts but rather to tackle the auxiliary challenges like dealing with large data sets, computational requirements and optimizations, and the deployment of models and data to large software systems.

Software 1.0 vs Software 2.0

Most classical software applications are deterministic where the developer writes explicit lines of code that encapsulate the logic for the desired behavior.

Whereas, the ML software applications are probabilistic where the developer writes a more abstract code and lets the computer write the code in a human unfriendly language i.e. the weights or parameters required for the ML model. Andrej Karpathy has written in detail about pointing to the same difference in his blog.

This requires us to look at new ways of getting data, cleaning, training, and deployments methods since apart from code, we have the weights and data which keep changing.

Extract Transform Load (ETL)

One of the first steps in starting machine learning projects is to gather data, clean the data, and make it ready for the purpose of experimenting and building models

The initial techniques may start with doing the above in a manual way but without automated pipelines to operationalize these ETL processes, the technical debt increases over time.

In addition, there needs to be a way to store large data either on cloud storage or file storage system. Storage also means proper tooling for gathering, labeling, and making the data access scalable.

Finally, as the data is being transformed, it is key to keep track of versions of data so downstream, when the data is being used for experimentation, training, or testing of algorithms, there is a trackable version of data that run can be associated with.

Machine Learning Experiments

Once data is gathered and explored, it is time to perform feature engineering and modeling. While some methods require strong domain knowledge to make sensible decisions feature engineering decisions, others can learn significantly from the data. Models such as logistic regression, random forest, or deep learning techniques are then run to train the algorithms.

There are multiple steps involved here and keeping track of experiment versions is essential for governance and reproducibility of previous experiments.

Hence, having both the tools and IDE around managing experiments with Jupyter notebook, scripts, and others is essential. Such tools require provisioning of hardware and proper frameworks to allow data scientists to perform their jobs optimally.

Deployment and management of model and ML pipelines

After the model is trained and performing well, in order to leverage the output of this machine learning initiative, it is essential to deploy the model into a product whether that is on the cloud or directly “on the edge”.

Deployment in Machine Learning systems can mean any of the following:

  1. Offline predictions: If you have large set inputs you would like to get the predictions on them without any immediate latency requirements, you can run batch inference in a regular cycle or with a trigger
  2. Online predictions: If you would like to make predictions soon after the request is made then this deployment helps while making calls either by serving as REST APIs or RPC calls.
  3. Edge deployment: You can perform online predictions while keeping on the device to decrease the delay in making online calls. This has a trade-off with accuracy vs power consumed

Following showcases architecture diagram with AWS tools. Either of these setups allows a data scientist or machine learning engineer to deploy their models.

Deployment and types of predictions

Monitoring of models in production

After the model has been deployed it is also essential we understand the performance of models in production to avoid issues with model or concept drift.

Tools for monitoring visualizations of data distributions and creating metrics around how the test data differs from the training data can be leveraged to track ongoing model performance and ensure the best performance.

Continuous Integration

Usual software continuous integration (CI) principles do not directly translate into the world of ML. Data scientists and ML engineers are not writing code according to a prototype specification, so it feels unnatural to write unit tests.

CI for ML has two key goals:

  1. Reproducibility: Ensure the key bits of code are working correctly and consistently
  2. Performance: Monitor progress and improvements in our predictions over time

CI should be built by using test data and running prediction scripts while validating the outputs received from the model on ground truth data at regular intervals.

Continuous delivery

This step combines all the above mentioned steps into a reproducible pipeline, which can be run sequentially. Generally, it will include the following steps:

ETL of data: Create a robust self-service data pipeline along with a set of utility tools that make the gathering, loading, transforming, and building datasets a much faster and simpler task

Continual re-training of your models

  1. ML service needs to be integrated with a customer-facing production feature, we need to be able to ensure we can sustain it through time. One part of sustaining a model is the ability to retrain it or recreate it from scratch
  2. Once finished, after training, store accuracy test results into a DB and the associated models in a File Storage system (e.g. AWS S3)

Establish a Continuous Integration pipeline around using test data and run predictions scripts to validate the reproducibility of the desired output and the performance of the model.

Then we can deploy the model into production either for online prediction as REST API, offline predictions, or have it deployed on the edge. In the deployed code establishing a method for ongoing monitoring of these models enables full end-to-end performance modeling of the model.

Continuous delivery ensures that a pipeline for all of the above steps is created. Traditional software tools for continuous delivery can still be leveraged with a few tweaks, along with specific data processing and machine learning tools to establish an automated and reliable workflow.

Artificial Intelligence in Practice

AI topics for AI practitioners by AI practitioners

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store