3 reasons you’re probably stuck in pilot purgatory
Machine Learning (ML) and Artificial Intelligence (AI) are increasingly common techniques, and brands are building their data science capabilities to solve their business problems more effectively. However, despite hiring new talent and building this capability, many businesses have failed to make any real business impact with their previous initiatives. It’s common for us to get our clients beyond the pilot phase — they have lots of initial excitement and anticipation about what data science can do, and lots of the ideas generated evolve into pilots, but far fewer get turned into something robust, operationalised and sustainable.
Below, we share some common reasons you might be stuck in pilot purgatory and what you can do to see through the last mile.
1. Data — start with what you have.
Machine learning models need data to learn from, and in many cases, this data will come from within many parts of the organisation and external sources. Finding and collating this initial data snapshot is hard enough and usually requires cooperation between business, technology and data owners most organisations aren’t used to. Many pilots fail here because they can’t get over this initial data hurdle. This requires engaged leadership who can ensure business and IT work together on value-led initiatives. It also means that the use-cases prioritised are ones based not just on value, but also on data availability and speed of delivery as a sequence criteria. It’s important to start with what you have and acknowledge the data that’s readily available. If you don’t have it — flag it early, put that use-case on ice for a few months, and more importantly — instrument your business so you will have the data ready when you revisit it.
2. Users and business change — no algorithm is an island!
Whatever solution you’re trying to build, it’s unlikely the algorithm will stand alone; business processes will need to change too. Whether it’s :
- a classification model predicting if a customer might miss a payment
- a time-series model forecasting SKU demand
- an optimisation model to determine the optimal price for a hotel room,
the model output needs to be consumed in a way appropriate business interventions can be made to improve performance:
- the late paying customer is offered a payment plan
- manufacturing pipelines are reconfigured to produce different SKUs
- hotel prices and promotions are updated.
These all require business process changes, and in many cases will require people’s roles to change too. All these users are unlikely to have been involved in the model development process and might need to be trained in how it works so you get the best of human and machine. These process changes can be significant — especially if they affect many employees and/or make a significant impact to your performance (which it should if you’ve chosen a high-value use-case!).
Thinking ahead to not only how a model will be used operationally, but by whom, is a crucial step to consider even before you embark on a pilot. There’s little value in building a pilot model if it can’t be used by anyone in your business. This is one reason DataSparQ brings a product mindset to developing AI solutions for our clients; something we will talk more about in an upcoming post.
3. Operationalising models is hard — inspiration vs. perspiration!
Building a good machine learning model is hard. Despite the availability of tools, technologies, libraries and Stack Overflow; identifying a meaningful problem which data science can solve, sourcing the data to solve it, and building a model that’s accurate enough is a complex task. This is why data scientists who have the ability and inspiration to do this are in such demand. However, this is only part of the puzzle.
We’ve already seen how bringing a product mindset can prepare for some of the process and operational change. But there’s another complex engineering part of the puzzle - making sure your model is not only accurate, but also hardened to be reliable, resilient and scalable when used operationally. Most data scientists are comfortable exploring data and developing models (in R or their favoured languages and libraries) in their preferred development environment (e.g. Jupyter notebooks running on their laptop) — but if your ML model is supporting business critical decisions (Which late paying customers do you contact? What SKUs do you manufacture? What do you price a hotel room?) it needs to be robust. Productionising models to make sure they not only remain accurate, but run fast enough, fail gracefully, can be supported, run cost-effectively, and meet all the other SLAs required, is hard. We’d expect more time to be spent operationalising models rather than developing them. Having engineers involved in your pilot to work alongside the data scientists is crucial to make sure your pilot can be productionised. After all, the only good data science code is production-ready code!
In summary, this is a team sport. Data scientists are a key member of any team to deliver these solutions — but they should be accompanied by Product Managers, Engineers and supportive leadership to make sure your pilot has a chance of making a real business impact. Once it’s productionised, you also need to consider the team that’s going to monitor and maintain it. One of our engineers shines a light on what this can involve in their article: Why your ML model is broken.
If you’re thinking about how data, analytics, ML and AI can help you work more efficiently, but don’t know where to start — join us for our webinar HERE, where we’ll share some tips on how to identify high-value problems suitable for AI. If you have a number of languishing pilots that failed to make the last mile — get in touch to see if we can help.
