Photo by M. B. M. on Unsplash

Forecasting in Times of Disruption

by Javier Recasens

The COVID-19 pandemic has radically impacted consumer behavior, creating enormous disruption and uncertainty in many industries’ operations. Companies have seen demand surge, plummet, or sometimes both, and the resulting confusion makes it hard for their supply chain teams to react.

When a highly disruptive event like this occurs, historical data might be insufficient, even irrelevant, for modeling the future. The consequences of this temporal break are far-reaching — time series forecasting methods, a mainstay of the forecaster’s toolkit, may no longer be a viable prediction option, as they rely on the assumption that the past is indicative of the future. But today, the future is unlike the past, which means forecasters need to rethink their approach.

This article contains our data science team’s collected thoughts on how to improve your demand forecasts in the wake of COVID-19. If you’re a fellow data scientist, check out these tips to help you support your organization in these challenging times.

Figure 1 — COVID-19 has dramatically changed consumption-based patterns (graph used for illustrative purposes only)

Measuring the Impact of COVID-19

The first rule of forecasting is to understand your data. In this case, that means understanding which of your forecasts have been substantially impacted by COVID-19 so that you can spend your time and energy judiciously. By quickly identifying problem areas, you can manually adjust your forecasts and put out fires in the short term, and prioritize your modeling efforts in the long term.

This process should include three main steps: find out where changes started happening, quantify a baseline for what should have happened without the disruption, and then calculate how far our baseline is from reality.

1. For each time series, detect the point where overstocking and panic buying began, also known as the “breakpoint.” The COVID-19 breakpoint is different for each item and location; in the US, it should be around March 1st, 2020 (several methods exist to find time series change points, but somewhere around 3/1 is fine to get started).
2. Compute a baseline that emulates normal behavior. The simplest baseline is last year’s demand (e.g. a seasonal naive method), but we can also assume our forecasts before the breakpoint are fairly representative of an undistorted world.
3. To quantify the impact of COVID-19, we can use year-over-year (YoY) metrics split out by category and aggregated by month. Weekly aggregations can also be useful, but start at the month level to limit the impact of noise in the data. Examine the difference in major statistics (e.g., mean, variance, and strength of trend) for periods before and after the breakpoint. (N.B. COVID-19 prevalence varies widely across the globe and changes by day. There are clear indications that the local magnitude of the COVID-19 effect matters for store traffic. US states that were hit harder will see bigger downturns in store traffic, while areas where it isn’t as widespread may show continued growth. Predicted COVID-19 trajectories can be used to calculate the lag between states.)

Forecasting Demand After a Sudden Disruption

Forecasting a target that’s dependent on a pandemic is very difficult. As explained by Rob Hyndman, the predictability of a quantity depends on several factors including:

1. how well we understand the factors that contribute to it;
2. how much data is available;
3. whether the forecasts can affect the thing we are trying to forecast.

The first two factors usually aren’t an issue for demand forecasting. Today, however, we don’t fully understand the drivers of demand, and in some cases, past data may no longer be useful.

COVID-19 Predictions

Forecasting the implications of COVID-19 plays a direct role in consumption-based forecasts. As long as social distancing and related restrictions are in place, demand patterns will be affected.

If you want to forecast demand, having COVID-19-related predictions as features in your model(s) could potentially enhance your results (such estimates are already available from a few different sources).

Figure 2: IHME’s COVID-19 projections were developed in response to requests from the University of Washington to determine when COVID-19 would overwhelm their ability to care for patients.

Forecasting Approach

Forecasting at scale is required for organizations with thousands or millions of time series (i.e., histories of item counts and locations for several levels of aggregation), often including extensive automated model selection, automated ensemblers, and more. But a fully automated machine learning approach may no longer be the right strategy for these turbulent times. Yes, you should include some of these auxiliary processes to make your forecasts accurate and useful, but the forecasting process should be prioritized. Resources are limited and speed is crucial (so serious that Amazon stopped accepting non-essential goods into their warehouses), so you should only focus on the most important products in your portfolio.

If accuracy is critical, we recommend the following model types:

• Deep learning-based algorithms that excel at short-term forecasting and cross-learning (i.e., they can pick up on relationships between time series) are excellent choices, especially LSTMs. When trained with external features and key organizational data (e.g., item and location attributes), these algorithms can detect trends and optimally weight observations if tuned properly. DeepAR might be a good choice for very big data.
• Statistical methods (e.g., structural time series models) that can model sudden changes in demand patterns have proven effective in similar situations. For example, Facebook’s Prophet automatically detects changepoints and adjusts model parameters to capture the newest trends in a time series. Users can also manually specify changepoints if they choose.

If control/interpretability is more important (e.g., if error metrics are high across the board), we would suggest these instead:

• Exponential smoothing (ETS) enables significant control over parameters that estimate the level, trend, and seasonality of a time series. They also generate forecast‐based formulas that can overweight recent observations.
• For maximum interpretability, we recommend simple models, like lift-adjusted seasonal naive methods. These approaches involve measuring change from the baseline (YoY change) at higher levels of hierarchy (e.g., category level), and then propagating this lift to the hierarchy’s lower levels. Measuring the impact of COVID-19 at different levels of the hierarchy may help you determine the optimal adjustment.

(N.B. We do not recommend using tree-based methods (e.g., random forests, gradient boosting machines), as tree-based methods perform poorly with data outside of the training data’s response range. Differencing can help these models extrapolate, but we remove the effect of the intense shocks we’re seeing by doing so.)

High-Demand Scenarios

If you just want to meet demand, a traditional point forecast may not be enough. For those cases, we recommend the following:

• Quantile regression can be used to forecast ranges of high demand estimates to help you meet acceptable service levels. Quantile forecasting involves modeling the entire demand distribution, and its flexibility makes it an ideal method when the costs associated with over- and under-forecasting aren’t the same.
• Stockout prediction models can help you determine the probability of a stockout over a given timeframe. This model type’s main objective is to identify items with a high likelihood of running out of stock, and can be implemented as a classification algorithm with the target variable as either a stock-out indicator (1 or 0) or an inventory level range (e.g., low/medium/high). Rate-of-consumption metrics, inventory levels, and demand forecasts can serve as inputs to one of these models.

Other Considerations

As you may have picked up, modeling in this crisis is hard. However, there are other complicating factors to consider controlling for during the modeling process:

• Ideally, our models should include causal features, relevant COVID-19-related metrics that might impact our target variable (e.g., lockdown prevalence and severity, local social distancing guidelines). Quick versions of this analysis could be as simple as comparing line charts of related factors with demand patterns, computing a relevant correlation matrix, or clustering time series based on these relevant external factors.
• Different regions of the US are in different phases of the COVID-19 life cycle. As some cities/states may be further along the epidemic curve, lagged features might help you predict patterns in other states.
• Since the situation is rapidly evolving, we recommend regularly retraining and re-running your impact analysis, preferably with daily or weekly frequency.
• Categorizing your items will be useful to determine different modeling strategies. For example, widespread panic-purchasing on essentials has pulled demand forward. Consumers aren’t actually using more bath tissue than before, but want to stock up for peace of mind. In situations like these, you should expect near-future demand drops as consumers use up their excess stock before buying again. Heuristics and manual intervention may be needed for these cases, as items rely on different consumption assumptions.
• In short-term forecasting, the very recent past is the best predictor of the very near future, and COVID-19 hasn’t changed that. Weighting recent observations more heavily than more remote observations could be helpful.
• We recommend using downstream consumption data (e.g., POS data) as the primary source for modeling whenever possible. POS data is a more direct proxy for demand compared to purchase orders or shipments. (N.B. Due to COVID-19, many demand transactions (i.e., consumer store visits) with zero sales of a given item actually reflect a lack of sufficient stock as opposed to a lack of demand. To forecast most effectively, you should try to estimate true demand the best you can, perhaps imputing sales with one of the multiple state-of-the-art imputation algorithms and substitution methods available.)

Overriding Your forecasts

If available, we recommend using tools that can enable forecast overrides (e.g., write-back solutions). Forecast overrides are adjustments that correct or include information not captured by the underlying model (e.g., sudden release of an OTC antiviral, an unexpected increase in social restrictions). Evaluating model accuracy after an override can be challenging, so we recommend saving the original, unadjusted forecast value for later evaluation. Having the flexibility to quickly include domain knowledge or other relevant external information can definitely be useful.

Manual adjustments aren’t possible for millions of time series, so forecasts for high-value categories (or other highly aggregated levels of the hierarchy) should be post-processed to include overrides and then propagate them down accordingly. For example, if you expect a 10% lift in one or more categories as a result of COVID-19, apply that override at the highest possible level and enable top-down proportional propagation to all related items.

In addition, what-if scenario analysis can also be used to test different demand and capacity situations to understand operational implications. For example, if your models use IHME’s COVID-19 projections, you can use the high and low values of its prediction intervals as inputs to more pessimistic and optimistic scenarios.

Conclusion

At least for the next few months, COVID-19 will continue to affect the patterns of many consumption-based time series. That being said, we will get through this eventually.

Whenever we do finally begin to approach a recovery point, we can try to detect it using changepoint identification techniques and pertinent statistical indicators (e.g., MACD). External data sources, like social media sentiment analysis or consumer sentiment indices, can also help us pick up on signs of recovery. Doing so will allow you and your organization to readjust more quickly to the post-COVID-19 world.

In the long term, the COVID-19 period will (hopefully) be treated as an outlier, but it’s unlikely to be the last crisis that supply chains ever face. However, if you continue to assess impact, adjust models accordingly, and use forecast overrides as necessary, you can find a way to get your organization through this crisis.

Many thanks to those who contributed, with special mention to Abhishek Nahar, Prasanna Ragavan, Keerthi Nallagatla, Hari Murakonda, Madhav Durbha, Sushant Khopkar, Amir Kalantari, Vikram Srinivasan, and Bradford Winkelman.

_________________________________________________________________

If you liked this blog post, check out more of our work, follow us on social media (Twitter, LinkedIn, and Facebook), or join us for our free monthly Academy webinars.