Machine learning short time to production: some advices

Article written with Thomas Vial and Matthieu Lagacherie

At the Total Digital Factory we have to manage a big challenge: fast deployment of machine learning (ML) in production. Indeed, our typical development time for a full stack MVP (i.e. including machine learning, data engineering pipelines, front-end development, etc.) is 6–8 months, developed by a squad of 8 people. Our development practices have to be straight to the point. As a consequence, here are some advices (not exhaustive)that I like to give, for data scientist and ML enginneers but also for product owners and business teams.

Photo by Guillaume Jaillet on Unsplash

Advices for the data teams

Pipelines first!

Do not consider the ML part of your product as a single feature of the product. Instead, decompose it in several features organized in a ML backlog, managed in parallel of the product backlog. The first feature has to be the setup of the train and inference pipelines. The performance is not important, a dummy model can be used. The objective is to build the contract between business, ML and the rest of the development team. Once the ML pipelines and the development pipelines are setup, even with a dummy model (prediction using the last known value, an average, a business heuristic…), you have an end-to-end demo to start interaction with the end users. As mentionned in this article, delivering machine learning solutions is so much more than the model: your product is the pipeline, not the model!

Manage your model (MLOps 101)

Once this is done, how to improve the model’s performance?

• Fix data quality issues (too big to be adressed here!)
• Do data science
• Get feedbacks as soon as possible. These two approaches can be used, together or not:

1. Shadow production monitoring: deploy the model in production and compare its results to actual data (but don’t use it in you application, just observe how it performs). There is no place like production to measure really the performance of a model,
2. End user feedback: deploy the model in production and let the users interact with it to collect additionnal labels or business knowledge. Important: the process has to be explained to the users.

The sooner you get feedbacks, the better: don’t be ashamed of deploying a simple model first (and if you’re lucky, the users may be satisfied with it)!

When you and the users are happy with a model, don’t forget to ensure it will evolve properly over time:

• Monitor the performance of the model in production ;
• Never stop collecting user feedbacks to improve your training database ;
• Define a relearning strategy (online, batch, regular, trigger-based, manual) ;
• Avoid hidden feedback loops and have a strategy to evaluate the real benefit of an upgrade of your model or of a new model (shadow production to compare the new model vs. the deployed one, A/B testing, etc.).

Model’s management is stronlgy linked to your pipelines. This is the foundation of Machine Learning Model Operationalization Management, also called MLOps. This website is certainly one the best reference to go futher.

Note: think about these production constraints at the beginning of your project, as it may impact the architecture, the data model, the design of the product…

Data science tips

When working on improving the model performance, here are some data science tips for your ML backlog:

• The ultimate modelling algorithm doesn’t exist, don’t lose too much time searching it. We are a Factory, not an R&D department: regarding our timeframes, you will probably get the best possible results with usual models from usual librairies ;
• Usual ML evaluation metrics (RMSE, MAE, AUC, F1-score…) don’t mean anything for the users. Try to translate them in business KPI thus you will know the real added value of your model ;
• Analyse the errors of your model, it will provide very valuable insights for improvements (ie.: top-k for classification, residuals for regression…) ;
• Very important: challenge your dataset. Check if it contains sampling bias, measurement issues and choose a cost function adapted to the specificities of the dataset ;
• Validate during the training phase (with a strategy adapted to the problem, eg. K-fold CV is not valid when errors are correlated ) — but this does not prevent you from monitoring the model in production — ;
• If you are stuck trying to improve the performance with data science techniques, consider coupling you model with business heuristics.

Advices for the business team

The data scientist needs you!

Don’t think the data scientist is a wizard, who will fix alone all your issues as long as he/she has data. Even more important than data for data scientist, is a close relationship with business representatives. Business have to be fully involved to work with the data scientist to build a model really useful from a operationnal perspective. Many items can only be validated with the users:

• Understanding of the data ;
• Model’s business KPI evaluation definition ;
• Performance expected ;
• Usage of the model’s output in the operational decision-making process ;
• Users acceptation of the model (trust, explainability…) ;
• …

Never forget that the model has to be built for you, users, you know what you need best and you are the expert of your local context. The data scientist will bring a powerful toolbox to boost the way you are using you data, but you have to pave to road with him.

Accept uncertainty

Be aware that results of ML are not deterministic. This means you have to accept uncertainty:

• Be realist when setting business expectations regarding the performance of the model. There is not use expecting a minimum model performance before a data scientist has had a serious look at the data ;
• The perfect model can rarely be guaranteed. Rather, think with the data scientist about the minimum level of performance needed to get value from a ML feature and start with this objective (usually it is better to deploy in 6 months a 70% accuracy model than 95% accuracy model in 2 years) — however, thanks to the MLOps approach, this performance is likely to improve over time-. This article provide valuable insights about the concept of “minimum viable performance” for AI product ;
• The expected performance is sometimes not achievable. You really know what you can get from data only when you put your hands on. Sometimes, even a skilled data scientist won’t able to deliver the expected results. Accept it. Discuss with him/her and adapt the modeling strategy and the business usage of the model on the performance that can be delivered by the data.

Accept change

Be aware that building a ML feature is not the same thing as building usual software. It is is out of usual imperative programming (if… then… else…) and development methods. Why?

• Rules are learnt based on historical data ;
• Rules involve uncertainty ;
• Rules will change over time ;
• Uncertainty decreases with more data and users’ feedbacks ;
• Rules can become wrong when structural changes occur and when entering new spaces.

Again, accept managing this uncertainty even when your product is live in production and work with the data scientist to check if human-in-the-loop approaches can be setup to enrich the product’s MLOps process.

To go futher on AI product management, I recommend this excellent guide.

Photo by John Schnobrich on Unsplash

After reading the above, you may have understood that data and business advices are two sides of the same coin: to ensure the success of the development, it is indeed essential that business and data people know how to work together and have a shared vision of what it means to build a ML software.