MLOps — Advocating Better Engineering and Operations in Machine Learning

What is MLOps? Understand how it is going to help you in building an end-to-end Machine Learning Pipeline

Samhita Alla
Dec 2, 2020 · 6 min read
Background Photo from Unsplash

Machine Learning (ML) has forayed into almost all principles of our lives, be it healthcare, finance or education; it’s practically everywhere! There are numerous machine learning engineers and data scientists out there who are well versed in modelling a machine learning algorithm. Nevertheless comes the challenge of deploying a machine learning model in production. Coding a machine learning algorithm is the tip of an iceberg. For a machine learning model to be deployable, configuration, automation, server infrastructure, testing, and process management have to be taken care of. In conventional software engineering, DevOps does the engineering and operations. It bridges development and operations seamlessly. Likewise, MLOps can be applied in machine learning to support rapid deployments in production.

About MLOps

DevOps is a set of practices to facilitate building, integrating, testing, and deploying the software reliably. It could be utilized to aim at continuous integration and delivery with minimal overhead of deployment in the development arena. A similar set of principles can be applied in machine learning to build a robust ML system.

MLOps also requires Data Engineering. Data is the bedrock upon which ML is built; data procurement, verification and feature engineering are crucial components whilst developing an ML algorithm.

Altogether, model management, data schema management, and model deployment constitute MLOps.

Challenges

  • Data scientists typically aren’t into building production-level systems; they primarily focus on data analysis and model development through experimentation, thus demanding the need for expertise in engineering and operations.
  • Clichéd ways of designing and developing algorithms isn’t the machine learning’s forte. Ideally, it involves experimenting with algorithms, features and hyperparameters. Filtering reliable results by various comparisons is quite essential.
  • A machine learning model has to be tested effectively to ensure its robustness in the presence of credible data values.
  • Deployment of an online machine learning model requires training on the web. Hence, effective strategies in validating results are to be in place.
  • Due to the dependency on the ever-changing data being pumped into a machine learning model, constructive monitoring of the performance numbers has to be put into practice.
  • Reproducibility is often difficult to achieve in machine learning owing to the diverse software environments, versions (release iterations) and data. Yet, it is a vital factor which helps in the continuous integration and delivery cycle to proceed smoothly.

Overcoming the above-mentioned challenges is significant in constructing an efficient machine learning system.

The Machine Learning Pipeline

  • Procuring the Data: Selecting relevant data sources to extract the necessary data points to be used by the ML model.
  • Analyzing the Data: Examining the data deeply to understand the diverse attributes and parameters. Understanding the data schema and devising data preparation and transformation strategies.
  • Data Preparation: Feature engineering constitutes data preparation. It involves data cleaning, removing data skewness, data transformation and data reduction.
  • Designing a Model: Selecting a set of suitable models that could be utilized in defining the algorithm.
  • Training a Model: Tweaking hyperparameters to achieve high performance.
  • Evaluating a Model: Testing the model’s accuracy on a test data set.
  • Validating a Model: Finalizing the baseline model to be used in production.
  • Serving a Model: Deploying the machine learning system in a suitable environment by setting up the infrastructure.
  • Monitoring a Model: Logging the statistics (performance, latency, traffic, errors) of the model. If there’s a degradation in the performance at any point of time, deciding on the required step to be undertaken.

Each of the above steps can be manual or automated. Again, manual MLOps has its disadvantages.

Manual MLOps

Also, model’s versioning doesn’t happen in an automated fashion which would create discrepancies in model’s reproducibility.

No Continuous Integration/Continuous Deployment (CI/CD) is ensured as the model is assumed to be static in a general sense where no frequent changes are introduced.

Monitoring the performance of a model doesn’t happen all the time which could lead to overlooking the model’s degradation statistics.

A Step Ahead — Automated MLOps

  • Transitioning from one phase to the other has to happen automatically, using automated strategies.
  • Data has to be validated properly to identify the skewness in it. If the data is in accordance with the expected schema, the next phase can be triggered, else, manual intervention is required.
  • The model should be trainable online using a fresh batch of data. It should be able to compare the current metrics with the previous metrics (baseline model) and choose the best model to be moved to production.
  • The development environment has to be available in the production stage to retrain and test the model.
  • Modularity has to be induced into the code for better sustainability of the machine learning system.
  • Monitoring the machine learning system has to happen at both, micro and macro level.
  • Additional components include Feature store and Metadata Management. Feature store includes the feature related data collected throughout the user’s interaction. It can be reused for training and serving. Metadata Management is the storage of metadata pertaining to the versions, time consumed, hyperparameters and statistics. This helps in easier anomaly detection.

On the whole, in the automated version, rather than integrating a trained model, the whole machine learning pipeline (training + serving + monitoring) has to be pushed to production.

CI/CD Orchestration

Use Cases

  • Automated MLOps (MLOps Level 1) is useful when there’s continuous training to happen in the production environment. Modifications come by and training has to happen online.
  • CI/CD MLOps (MLOps Level 2) is useful when the updates come by regularly and a huge infrastructure is underlying the machine learning system.

MLOps Tools

Conclusion

Thanks for reading!

References: Google Cloud

Feature Stores for ML

AI, Data, and everything in between