Eurybia: MAIF releases a new open-source solution for quality IA models in production

MAIF, a French mutual insurance company (est. 1934), has been contributing to open source for a few years. Since 2019, we share our work in the field of artificial intelligence, our first release was Melusine, for french langage email processing and our second was Shapash, to make machine learning interpretable and understandable by everyone.

We are now releasing Eurybia, a python library to ensure quality of machine learning model in production by detecting model drift. Eurybia also addresses challenges of industrialisation and maintainability of IA over time. Thus, it contributes for a better model monitoring, model auditing and more generally AI governance.

Eurybia logo by MAIF

Elements of context

More and more machine learning models are being deployed into production environments. A key step for deployment is to secure the data that will be used for the prediction model. This is crucial, notably when training and production data sources differ.

While most models have been built from historical data and are static, data are usually dynamic, meaning that they change over time and so do the prediction capabilities. To have a controlled use of AI, it is important to detect these changes to determine if they are problematic, track performance over time, and consider re-training the model if necessary.

Why have we made Eurybia?

At MAIF, as an insurer, we’ve been using data for a long time, notably to simultaneously offer the best personalized advice to our customers and control insurance risks. Because of our history, as a committed mutual insurance company and because we believe that companies should be responsible for their use of data, we are particularly involved in AI governance.

The visualization of the life cycle of a machine learning model can ease the understanding of Eurybia importance. During their life, ML models go through the following phases: Model learning, Model deployment, Model monitoring.

life cycle machine learning model

To ensure that models perform in production as well as they did in training, it is important to verify that data are consistent and that features distributions are similar.

We want to closely monitor our models in production, notably to detect as early as possible when models are likely to provide less suitable recommendations.

At MAIF, we use Eurybia to:

• Secure model deployment by detecting any inherent data problems
• Help data analysts, data engineers and data scientists to collaborate for data validation before deploying a model into a production environment
• Make it easier and faster for MAIF’s data scientists to analyze data drift
• Know whether it is necessary to re-train the model
• Save drift monitoring reports for drift monitoring, for consultation following an e-mail alert or another specific need

What is Eurybia?

Eurybia is a Python library which aims to help detecting drift and validate data before putting a model in production (named deployed model).

Eurybia Gif

Underneath, you’ll see a short description of elements which are referred to in this article.

• Data Validation : Validate that data used for production prediction are similar to training data or test data before deployment.
• Data drift : Evolution of the production data over time compared to training or test data before deployment.
• Model drift : Model performances’ evolution over time due to change in the target feature statistical properties (Concept drift), or due to change in data (Data drift).

For technical definitions, we recommend to read the bibliography.

For Data Validation and Data Drift, Eurybia works mainly with a binary classification model (named datadrift classifier) that tries to predict whether a sample belongs to the training/baseline dataset or to the production/current dataset.

How the datadrift classifier works

The explainability of this datadrift classifier provides higher scores to features that are important for drift. When combined with the initial deployed model features importances, this explainability reationalizes further analysis.

Eurybia thus allows its users to :

• Get a quick overview of features that changed the most to prioritize study of drift
• Put into perspective the evolution of features with their importance for the deployed model. To study in priority drift on features with the most influence on the model
• Display each feature and its distribution
• Display the difference between the predicted probabilities distributions respectively based on the baseline dataset and the production dataset
• Have a performance metric for data drift and track it over time

Eurybia also has additional features that allow to :

• Analyze data consistency by checking if features are the same between baseline and production dataset, as well as features modalities
• Detect model drift by displaying model performance over time (a section we would like to develop in future Eurybia releases)
• Save a dynamic report that allows collaboration between colleagues, drift analysis, and for audit purposes

Eurybia can be used for all kinds of use cases and domains : health, business, marketing, …

Please visit Eurybia’s GitHub : there, you will find more exhaustive documentation of the library’s features as well as a demo of the dynamic report for a quick test! Please do not hesitate to give us a star on GitHub if you like the project!

Maif open source several libraries and not only data science libraries, you can visit them here.

Further readings

We have quickly introduced the notion of drift. If you wish to learn more about drift, I recommend the following resources :

• Drift classification and retraining: https://towardsdatascience.com/model-drift-in-machine-learning-models-8f7e7413b563
• Concept drift characterization: https://arxiv.org/pdf/1511.03816.pdf
• Mathematical view for different types of drifts : https://medium.com/data-from-the-trenches/a-primer-on-data-drift-18789ef252a6
• Data drift detection: https://arxiv.org/pdf/2012.09258.pdf