Managing Predictive Systems During An Unpredictable Pandemic
The COVID-19 pandemic has perplexed data scientists as the sudden changes in consumer behaviour has made predictions based on historical data very difficult. While these challenges could shake our perception of what machine learning is, they might also foster the development of tools that could automatically adjust.
When it comes to predicting demand or consumer behaviour, there is nothing in the historical data that resembles what we saw during the COVID-19 pandemic. Thus, models based on historical data tried to reproduce what is normal and failed to give accurate predictions.
Let me give you a simple analogy of this problem. If you want to predict how long it will take to drive from A to B in London next Thursday at 18:00, you can look at historical driving times at various scales. For instance, you can look at the average speed on a day at 18:00 or on a Thursday versus other weekdays. The same reasoning can extend to time scales like one year, ten years or what is relevant for the quantity you are trying to predict. This will help predict the expected driving time under normal conditions. However, if there is disruption on that particular day, like a football game or a big concert, your travelling time might be significantly affected.
Perhaps unsurprisingly, machine learning tools deployed across various industries — from transport to retail — struggled in coping with the massive changes in the behaviour of both users and the environment. While one can focus predictive algorithms on recent data, one cannot expect regular outcomes and the same quality of predictions as before.
What to do?
While data science solutions are built on long-ranging historical data, we should continually assess their performance on the most recent data. Extraordinary data coming in with consistent drops in performance strongly indicate that the rules have changed (a phenomenon that is also known as dataset drift).
For now, performance monitoring is independent of the systems — it tells us how things are doing, but will not change anything. However, we are seeing a push towards systems that could adjust automatically to the new rules. If we can make a system adaptive, then it could improve itself based on the recent data when it identifies performance drops.
Building these adaptive systems is still aspirational. Nonetheless, to cope with the unpredictable landscape of the pandemic, practitioners adopted other techniques such as:
- Analysing the model and understanding how the new circumstances influenced the predictions;
- Training predictive models with potentially similar data from a previous crisis with similar characteristics;
- Updating training data frequently, quickly iterating and improving models on the most recent data generated during the novel circumstances;
- Reducing the timeline of training dataset as much as possible without altering performance metrics; and
- Removing data from the pandemic period from the training data upon confirming that behaviour returned to pre-pandemic levels.
Beyond the technical aspects, disruptive periods require careful management of stakeholder expectations, especially of users’ and clients’. Developers and managers that managed the challenging times overly communicated that noise and novel behaviour made it difficult to maintain accurate predictions. Also, challenging periods required setting KPIs more flexibly.
Clearly, building a model that can respond to extreme events incurs in extra costs and is not always worth the effort. However, if you decide to develop such a model, focus on capturing long- and short-term patterns of your data. Assigning distinct weights to long- and short-term information will enable the model to adapt more sensibly to extreme changes.
In the long run, this crisis reminded us that there are events so complex that even we still struggle to understand, let alone predictive systems. We too need to adapt to the new normal and update our internal parameters. For example, to better forecast how long it will take to do the weekly shop or choosing a new optimal path when walking down the street.
This adaptability is natural for us humans, and it is a feature we should be trying to impart on silicon-based intelligence. Ultimately, we need to recognise that an AI solution can never be seen as a finished product in the ever-changing and uncertain world in which we live. How we enable AI systems to adapt as efficiently as we do is still an open question. The answer will define how much our technology will be able to be of help during volatile times that might be ahead of us.
I thank my colleagues Alex Lilburn, Ted Lappas, Alistair Ferag, Sinem Polat, Jonas De Beukelaer, Roberto Anzaldua, Yohann Pitrey and Rūta Palionienė for providing insights and helping me to prepare this article.
