Fears in switching to AI/ML demand forecast — A user point of view
Doubt, Challenges and Fears of a Demand Planner/End User
I had the opportunity to lead the AI/ML based demand forecasting product build for a Fortune 50 retailer. Of the many learnings, the biggest one is why AI/ML adoption is tougher and what things make it less tough. Every change is a challenge. AI/ML is no different, in fact a little harder because of its relative newness.
In this blog, I attempt to replay what I have learnt as fears and challenges from a end-user (in this case, demand planner perspective), where these fears are rooted and what product leaders, innovators need to be intentional about.
The Role of a Demand Planner
Traditionally, a demand planner is expected to maintain a “good” forecast, provides input to position “optimal” inventory at selling and fulfilling locations. A “good” forecast is usually measured based on forecast accuracy metrics like Mean Absolute Percentage Error(MAPE), Forecast Bias. Creating a “good” forecast includes using a model created forecasting and layering on a host of additional insights about location, weather, product, pricing, competitor, market insights. Additionally, demand planners collaborate with store operations, regional teams, internal merchants and external vendors for inputs, insights that demand adjustment to the forecast.
Demand Planner Lacks Trust in a “black box” forecast
Historically, Time Series forecasting is what demand planners (folks who are responsible to create a demand forecast) have used for decades. The time series forecasts provide parameters, controls that planners can change to influence the forecast number. I relate to them as co-efficient in an equation that will change the output value. In most companies, many demand planners have a degree in statistics and/or have learnt that as part of their job.
Come to AI/ML forecasts, and the way they work (and the ones I have seen), there is little influence/co-efficient like changes planners can do to change the model output. Also, given its newness, and limited or lack of forecast “explain-ability”, demand planners feel it is a black box with a lot of unpredictability.
Demand Forecasting is someone’s entire job
Traditional demand planner roles required planners to constantly adjust parameters, manually upload/edit forecast numbers. It was their entire day job. Not only that, but the forecast’s accuracy is also their performance metric. And now it makes sense for someone to feel uncomfortable about operating with a black box based on which their individual performance is calibrated.
The comfort factor (with current system)
By virtue of long-term use, demand planners know what works and what falters in time series/current forecast models. They know how to “manage” the forecasts. Keeping aside how good or bad time series models are, demand planners are USED TO it. “Being used to” something for a long time is most often results in increased resistance and fear for change.
Business Leadership Fears vs End User Fears
Business leaders typically provide buy-in based on big picture — trends, overall industry direction, “projected” potential benefits. Most often business leadership fears are about precise and disruption-less execution. On the other hand, when it comes to end users, demand planners, the challenges and questions are very specific Ex. Forecast approaches and controls for bulky products stocked in upstream nodes need to be different form store stocked products, forecast number need to be explainable.
Questions from end users from real life AI/ML demand forecast rollout:
- How do I trust that AI/ML forecasts are better than what the current time series forecasts are, is there proof that the results are consistent across a longer time horizon, could you prove that the forecast improvements at a smaller scale, however explain the company level impact?
- Is there stats to prove how forecasts for my product category are better for various seasons?
- What controls do I have over the forecasts? If the model repeatedly does not get a forecast right, how can I “make” the model do what it is supposed to do?
- How can I educate the model about my product categories or factors that the model might not know? What if last week’s sales in South region were because of a one-off local event?
- How do I know what inputs were considered in model and how much each input was weighted?
- How do I know if forecasts go wrong?
- How will the AI/ML forecasts continue to support downstream supply chain planning. Ex current time series forecasts predict base demand and that feeds into safety stock planning Ex. Promo forecasts drive purchase ordering (in advance) against promo demand?
- How do I explain a forecast number in my weekly supply chain inventory review meeting? What is the breakdown of the total forecast? Of the total forecast , what is — recurring demand — trend, recurring demand — seasonal, what is non -recurring demand — one-time promotions, weather, one-time local events?
- If the forecasts go wrong, who is accountable — the AI/ML data science team, the demand planner or both? Does that necessitate the data science team is also represented in the business meetings? Or do data scientists also double up as demand planners?
- Are my measures of success/performance metrics the metrics that forecasts also try to align on? If my forecast metrics targets change for diff kinds of product over different periods of time, to support stock up or clear down inventory strategies, how will the forecast adjust?
- How do I know that the variability in forecasts over time does not result in excess inventory or out of stocks ? What is the variability of the forecast between ordering and replenishment. If the forecast is 10% positively biased at ordering and negative 10% bias at the time of deployment, it causes on hand inventory (excess inventory) to go up by 20% . And the reverse is also true and could lead to Out of Stocks?
- In the AI/ML forecast world, what is the new role of the demand planners? What do you expect me to do, on a daily and weekly basis? What is the role of the machine? What is the role of the human?
Setting up an environment for open user feedback is critical for success
As a practitioner the biggest lesson I learnt is to foster an environment and create a open, unbiased channel to allow the user’s feedback, questions and concerns. Missing to set up this sort of a culture and discipline leads to significant delays in rollout, adoption later in the product journey and leads to uncertainty in the future of the product.
Few ways to set up open lines for end user engagement and feedback:
- Listen with Empathy and Respond with action. Internalize the questions/concerns/feedback. Ask key users (not stakeholders) for what they need. Sieve through the feedback, send responses, communicate action plans. If answers are not available right away, provide approaches. The answers need to reflect in the product strategy and roadmap.
- Form a steer co comprising key business and tech stakeholders to brainstorm key issues like change management, process blockers, demand planner role clarity etc. A democratized facilitation and a diligent follow-up is key to making the steer co truly work its purpose.
- Involve end users (not just business leaders/stakeholders) involved in the evangelization process. Continue to make sure that users are constantly involved in the roadmap and vision. Users should feel they have an input into and are part of the product build process and not just users/consumers of a product. As a response users will most often also tell what they don’t need.
- Form a core team of super users, who have a detailed say in the product features. This team is also the beta user team who tests the product feature and gives early feedback before larger investments are made in a feature.
- Choose a champion for adoption and change management. A champion is someone that understands the larger need for the program but someone who the users trust as their voice. It might be better if the champion is from business, and most importantly should be someone that both business and IT teams trust as the success agent and spokesperson.
- Make user engagement a culture and a habit. An example could be through a) Design workshops, product workshops with end users early on and repeated at a quarterly cadence b)Open channel for feedback — a mailbox, a slack channel a comment option or anything that any user could send a question, concern, or message to. This option works very well for users who have a high resistance but reluctant to voice their concerns in a group setting. After the product has reached a certain stage of adoption, the feedback could additionally include super users and business leaders as well.
Summary — End users/Customers decide success of AI/ML adoption
No concept, product or business initiative is successful until end user/customer is satisfied. Listening to user concerns, feedback is even more critical especially in a field like AI/ML that is relatively new. Given the potential for AI/ML, it is important that product leaders feel responsible to think of ways to build adoption on end user/customer satisfied.
Related Stories
