Our model factory serves 5 million members daily and is run by only 2 FT positions. Here’s how.
When someone asks me what I do for a living, I always reply with: “I’m modelling”. And then a pause.. The looks you get are priceless. Luckily I think people tend to be more impressed by my narrative tour through the prediction model factory of the ANWB than they would be if I were actually on a catwalk. Or perhaps that has something to do with the people that approach me. Anyhow, our prediction model factory in which we built over hundred models for personalized marketing is worth the applause.
Prediction models for personalized marketing
The ANWB is, besides being an information authority for its 5 million members, a provider of a wide variety of products and services. In addition to our famous road assistance service (‘wegenwacht’), we offer insurance products, packaged vacation deals, as well as many travel-related products through our physical locations and our web shop. Since we recently even became an energy-provider, we now have around 250 propositions to offer! In our owned marketing channels we use personalized marketing in order to show the most relevant proposition for every single member. To determine which proposition is the right proposition to show for an ANWB-member, we use prediction models.
How do we build a model?
Since we have so many propositions, but only 2 FTE, we cannot build a custom-made prediction model for every proposition. Therefore we streamline the model building process in the software KNIME (the Konstanz Information Miner). KNIME is an open-source data analytics, reporting and integration platform. The software integrates various components for machine learning and data mining through its modular data pipelining “Building Blocks of Analytics” concept. Besides writing a very standard ‘target-script’, which defines what behavior we want to predict, the model building process is done in this software. In KNIME we have a template that enables the opportunity to build a solid classification model within half a day. This template includes sampling, model training, testing and validation steps. Even the deployment is automatically handled. All our models are made with the same template, with no exceptions. In this way you don’t build custom-made models that could be close to perfect, but our models perform well enough to determine the most relevant proposition for an ANWB-member. It’s fast, convenient, and a very efficient use of our resources. But the best thing: You don’t even need all the skills or knowledge of a programmer to build a prediction model.
Prediction model factory
Next to the template for building a prediction model, we automated many other processes for our ‘factory’. First of all, scoring all of our prediction models is an automated daily process. The output of this process is a giant table of the scores of every member for every model. These scores are then input for some formulas to determine what is relatively the most relevant proposition to show to every member in our channels. Also this table is used by database marketeers for selecting relevant members for their 1-on-1 marketing campaigns, in channels like email. For them, it’s way more efficient to select the ten thousand members with the highest scores for a certain product, than to come up with their own rules and statements to come to ten thousand relevant campaign recipients, for example. Second, the evaluation of our prediction models is done automatically in KNIME. In this process we determine how much the Area Under the Curve (AUC) of every model would change, based on all the new data since we last trained the model. Based on this evaluation, it is automatically determined whether a model needs to be trained again. If so, this is also automatically done in KNIME.
How good are our models?
Our goal is to make many ‘good’ models that are automatically scored, evaluated and trained, rather than having ‘perfect’ models which require way more resources to maintain. Still, we developed models that predict better than we had ever hoped for. Not only do our nearly 100 models score an average AUC of 0.8, analysis showed that our models manage to correctly rank the right member for the right product at the right time. Members that fall within the top 10% even show a conversion rate of over 5 times higher than average. With models this good, we can perfectly serve five million members daily, with only 2 FTE.
Would you like to know more about our prediction model factory? You’re always welcome to get in touch via mvankoeverden@anwb.nl!
Author: Merijn van Koeverden (Marketing Data Analist)
