Data Science in Business
Concept Drift — a hypothetical case study
Introduction
Many say that the impact of Covid-19 will “change the world forever”, while others are trying to bring the world back “to normal” and to restore a thriving economy.
How does that affect data science? Surely, the models and methods we use are “scientific”, that means they use maths and therefore produce quantitative outcomes based on logic and data, they don’t get sick, take time off, get stressed from working at home while there are children about — if one thing is reliable in these crazy times, it’s our machines, right?
As my colleague Patrick Robotham outlined in his article on concept drift, the ways data can change are manifold and subtle. Most often, the data changes because its causal processes change.
This scenario is an exploration of how our machines can be fallible and what is needed to address the problem.
Changing Behaviour
During the isolation required to handle the Covid-19 pandemic, many businesses closed, including offices. This shifted the consumption of energy both in where, but also when it was most in demand.
The Situation
Let’s take an imaginary energy provider, Ethereal Energy. Lara is a supervising business analyst at Ethereal Energy, and her team is in charge of billing. Some customers have smart meters, so their consumption is available to both the customer and the retailer in near real-time. But customers without a smart meter have to rely on company estimates of their consumption until their meter can be read. As the restrictions around the spread of a pandemic similar to Covid-19 lift, Lara’s field operatives can visit their clients again and read the meters accurately to update their bills. Often, the predicted consumption is much lower than the value read by the field agent and Lara’s team is beset by angry customers. Ethereal has a large portfolio of renewable energy and has positioned itself in the market as the underdog, a fair and ethical provider on the side of the customer and now its customers are upset. People already have to make do with less, and now their electricity bills hit them out of the blue. Desperate phone calls, frustrated eMails and raging social media posts make life hard for the billing team.
Lara realises that the changes coming with the pandemic are not limited to the regulations being lifted. There are predictions of secondary waves, speculations about lasting changes to economy and culture and she finds herself worrying about the billing estimates failing so many vulnerable customers again.
She consults with her supervisor and arranges for the data science team to look into options to make the estimates better. One of the data scientists, Sienna, is meeting with her to discuss the current estimate modelling and future options.
Initial consultation
Sienna has prepared an overview of the current model used for the estimation of consumption by the customer.
“It’s a very straightforward linear model,” she explains. “It has some added features for the change of seasons, but it falls apart when things change as widely as they did in the pandemic response.”
“Right,” Lara nods. “So the model isn’t broken, it’s just not quick enough? Can’t we just, you know, speed it up?”
Sienna smiles.
“Not quite. It’s not the speed of the model, it’s the speed and type of changes. If you input for example the temperature, because you’d expect people to put heaters on. But because of the pandemic, people stay home and put the heaters on at different times of the day. And not just their heaters, they make tea — and they bake. I saw your sourdough loaf on the internal channel, it looked pretty yummy.”
Lara laughs.
“It is! Yeah, we’ve been cooking a lot.”
“So people are using much more electricity at home at the same temperatures as before, where they’d be in the office, using the office kettle and sandwich press and so on. The model can’t know that.”
“Right. I see.”
Lara frowns.
“But this is all meant to be AI now, isn’t there some artificial intelligence thing you can do? You know, add another input for how things change?”
Sienna sighs.
“I think if it was that easy, we’d all be sitting at the beach sipping cocktails with machines doing all the work. I mean, even groups of human specialists have a hard time predicting the effects of this change, and our machine learning algorithms just aren’t that far yet. If it’s something we’ve never seen before, they don’t stand a chance.”
“Okay, so what can we do instead?”
“We can take the new data we have and retrain the model.”
Lara looks at Sienna.
“That sounds like there’s a ‘but’ coming.”
Sienna sighs again.
“Yeah, there is. It’s not as easy as taking the new data and everything will be as before. Things are still changing — restrictions are lifting, so we know the new data won’t be accurate for long. But no one knows what’s coming after and how people will behave. Not to mention that with the short period of time since the pandemic, we don’t have a lot of data to start with.”
“Can’t you make the training part of the model? Or just retrain it every 3 months or so?”
“Yeah, we could. We’re retraining the model right now with the data we have to address its current issues. The thing is, we don’t know how much and when we need to retrain yet. We can’t just set a timeline and waste time and money.”
“Upset customers also cost us money, and we’ve got to address this. The future seems pretty uncertain at this point. How about this: I’ll talk to the risk team and check with accounting and marketing, see how they estimate customer reaction to the challenges ahead. I should be able to give you a price you can work with. And you look into options to retrain the models, costs and requirements and how to future proof this thing.”
“That sounds good to me,” Sienna agrees. “Let’s do that.”
Modelling Options
A few weeks later, Lara and Sienna meet again to discuss their findings.
Lara has spoken to her team and the risk and marketing departments to get an estimate for the impact in reputation and possible loss of customers. She forwarded the numbers to the data science team and Sienna and her team have prepared a range of options to prevent the model results from upsetting the customers again.
Lara is rubbing her hands.
“Alright, what have you got?”
Sienna smiles and brings up a slide.
“So first of all, as you know, we’ve deployed a new model already to address the issues we’ve had with billing before. It is working. For now.”
“Oh, start with the bad news, please,” Lara tries to put on a smile.
“We’re monitoring it and comparing it to a few test setups and it’s already deteriorating.”
“That sounds… bad? Is it going to break?”
“Well, it’s a slow change. It’s just losing its accuracy as the restrictions are loosening more and the new legal frameworks around prevention are put in place.”
“Okay, let’s look at the upside. That sounds like we were on the right track about the retraining — what did you come up with?”
Sienna straightened and forwarded to the next slide.
“We’ve got two options: We can keep running a model until we reach a threshold to retrain it — that could be the deviation from predicted billing, or number of customer complaints, we’ve used the numbers you sent me to establish and test this, it can be reported and triggered automatically. Our second option is a bit more expensive, but more future proof. It involves a continuous machine learning pipeline. We’d have a data silo that continuously gathers information and an integrated system to train and deploy a new model regularly to be up to date with the latest developments.”
“Sounds good, what are the pros and cons here?”
“Well, the triggered retraining is cheaper. The alerts can be set up automatically, but it might not be fast enough. Because it’s reacting to a trigger that’s based on something out of the ordinary, it could be that it’s behind the curve when things are changing fast. The machine learning pipeline has a much better chance of catching that, but it incurs some continuous costs because it is always on. But when things change, it changes with them, so it should soften the blow of quick changes significantly.”
Lara eyes the screen.
“I see you’ve got estimates for required development and maintenance in hours as well as provider costs on there. I can take this and present it upstairs. That’s great work, thanks Sienna.”
“You’re welcome, Lara.”
Presenting Upstairs
Lara has a meeting with the senior stakeholders of the billing and customer service representative departments to discuss options for how to deal with the estimate modelling.
She has prepared a short set of slides using Sienna’s numbers and projections about the model effectiveness and customer retention based on the more accurate estimates.
She concludes her presentation with a summary of the two models.
“So, in short, we have a model that retrains as needed, triggered by specific events, such as the model failing beyond a specified dollar amount too often. This option is cheaper and does not require any in-between maintenance, but it only kicks in when things are already changing and damage has already been done.”
She triggers an animation that highlights the other half of the slide.
“On the other hand, we have a continuous training pipeline. It requires a little more set up than the triggered model, but it retrains constantly and can prevent and buffer some of the impact of changes like the pandemic restrictions, most of all prevent customer loss. For that, it has an ongoing cost.”
She brings up a graph with several projections.
“I’ve been checking with the risk teams about the impact the first failure of the billing estimate model had. These graphs are showing how much the options cost compared to the cost incurred for us if circumstances change again, with three options for a worst case, a conservative estimate and a smooth, low-impact case.”
The model strengths are clear from the graph, in the worst case scenario, the machine learning pipeline is expected to buffer much of the change and save much more than its initial and ongoing cost. In the conservative estimate, however, the cost from incurred damages and triggering retraining are still lower than the total savings and ongoing costs for the pipeline model. In all cases, doing nothing is an unacceptably costly baseline.
She leaves the meeting set to presentation with the slide up and checks the chat for questions. There are no comments in the chat and the meeting is quiet.
Mark, one of the customer service directors, speaks up first.
“So, we’ve got the choice between these two, that’s what you’re saying? There’s no alternative?”
Lara clears her throat.
“Yes. These are the most cost-effective methods.”
“This is a good overview, Lara. And it’s scary as hell.”
Mark laughs.
Lara scratches her head.
“Thanks, Mark, I guess?”
“All good, Lara — you’ve gotten onto it quickly and thoroughly, that’s a good thing. What would you do?”
Lara swallows and takes a deep breath.
“I think customer retention takes precedence over immediate cost. We can control cost, we can’t control the social media message. My team has been drowning in angry customers. I’d take the pipeline.”
Mark nods.
“Fair call, I can see where you’re coming from. I guess that leaves the rest up to us. You’ve shared the slides with us all, we’ll review your proposal and get back to you.”
“Sounds good, thanks Mark.”
“Thanks Lara, see you.”
One by one, the senior stakeholders leave the online meeting.
Lara closes the browser tab and leans back.
Conclusion
A few days later, the senior stakeholders email Lara with a few more questions about details and after some clarifications agree to a trigger based model with a low threshold for a test period. If the trigger doesn’t appear to be sensitive enough, they will reconsider and look at options for a machine learning pipeline.
Lara informs the data science team and her own crew and everyone is relieved that they are doing something to be prepared for the future.
“The only constant thing is change,” Lara thinks as she jots a few notes down in her journal. “and even change changes.”
She archives the slide deck and considers her conversations with Sienna.
“Their machines have moving parts,” she thinks. “and they need maintenance, too. If the wrong data comes in, it’s like pouring sugar in the tank.”
She closes her laptop and takes another sip of coffee. Images drift through her mind of changing inputs, different beans in her coffee, different milk. Different paper in a printer.
“The machines aren’t clever enough yet,” she shrugs. “But that, too, will change.”
If you have any questions — aren’t sure about your models, want to future-proof them and want to know about the most cost-effective methods to do so:
Talk to us. Get in touch with us at Eliiza. We’d love to help.
