Why AI/ML Projects Fail — and How to Fix Them
From starting small to identifying use cases, there are various ways organizations can encourage successful AI programs.
Many companies are rushing to build machine learning (ML) and artificial intelligence (AI) solutions only to run into issues that cause their projects to fail. The failure may occur for many reasons, including:
- Lack of an appropriate use case.
- Absence of the right data and foundational pieces to access.
- No executive support to deploy a model into production.
- Unsuccessful efforts to scale solutions beyond early pilot work.
Fortunately, there are a few practical steps that organizations can take to avoid disappointments and failure in their AI programs.
1. Determine whether your business challenge can benefit from AI/ML
A common reason for failed AI projects is an implicit, unexamined assumption that the problem is best solved with ML. A surprising number of business problems are better addressed with simple automation techniques or explicitly codified rules. Still, with many companies now having their own machine learning departments, there is an increased risk of new AI/ML initiatives created based on a “hammer looking for nails” approach.
To avoid wasting money with AI, the first filter to apply before proceeding with a project is to validate that the problem can truly benefit from ML. Some companies may need external help during this step if they don’t have internal capacity, but skipping it creates a considerable risk of unsuccessful AI investments.
2. Establish how ROI will be measured
Once a valid use case for AI/ML has been identified, the next step is to define how the return on investment (ROI) will be measured. In many ML projects, it’s appropriate to adopt a classic ROI approach by asking:
- What is the cost involved?
- What is the value to be extracted from this initiative?
There are situations where the direct dollar value will be a secondary consideration. For example, I’ve developed ML models to help nonprofits optimally allocate scarce resources such as staff time, counseling, financial and legal resources to increase the likelihood of positive outcomes for the populations served. Desired outcomes in that scenario could be preventing underprivileged students from dropping out of school, or keeping families from becoming homeless due to house evictions.
In either situation, a project should only start after the company has a clear answer for the following question, with a tangible, quantifiable measurement behind it:
If we build a model for this, what improvement do we expect to see in “X?”
“X” should be an important dimension of performance, such as customer satisfaction, social impact, or environmental conservation.
3. Manage risks from day one
Even if a problem can be solved with AI/ML and has a high expected ROI, an ML solution may not be worth the deployment and maintenance costs or the potential downside of an algorithm making the wrong decisions.
When considering replacing a solution that currently relies on human intervention or a rules engine, it’s critical to quantify the downside of adopting an AI solution. Depending on the use case, the drawbacks may include anything from unfair treatment of underrepresented groups to customer dissatisfaction to a negative environmental impact.
Here are some steps to minimize risks:
A. Start with a modest project where the cost of being wrong is manageable and some tangible proof of ROI can be quickly established.
Small projects with clear metrics you can track are an excellent way to minimize risks, show that you’re fighting a winnable battle, and justify larger investments in your initiative.
For example, rather than attempting to replace an entire medical claim adjudication system with AI/ML, focus on a small, low-risk piece of the problem first. As a first step, ML could be used to classify claims as requiring or not an authorization so claims can be efficiently routed to different human-in-the-loop workflows.
Or — rather than kicking-off an “AI-powered digital transformation” with a chatbot to handle customer inquiries and running the risk of reputational damage if it can’t capture nuances in human conversations — start with an intelligent agent that simply writes support tickets based on the content of customer emails.
B. Timebox your efforts
Many AI/ML projects keep going for years without delivering any concrete value. The AI projects more likely to succeed are structured in phases separated by stage gate reviews: go/no-go decision points where appropriate stakeholders are brought in to review progress and agree upon continuing to the next phase.
Early in the effort, gates should include answers to questions such as:
- Do we have sufficient support from leadership to adopt a valid solution? If you can’t get the appropriate decision-makers behind the project, there is little value in spending time and money building an accurate and reliable model that later will face unsurmountable obstacles to be put to use.
- Can we deploy a winning solution? It’s useful to begin with small proof-of-concept projects, but it’s critical to understand from the get-go how to close the gap between the ML capabilities supporting a pilot and the capabilities required to enable scaling it.
C. Start your project with real, messy, live data
Many organizations start projects with very sanitized, simplified data. Then, after expectations have already been set with the model tested in a sandbox, they experience great disappointment when the solution fails to generalize to the real world.
For instance, while creating a proof-of-concept for a computer vision model to diagnose brain tumors based on radiology images, it may be tempting for data scientists to use a readily available set of high-quality MRI images taken from patients of a specific gender, ethnicity, and age group. Later, when the solution is finally tested with more diverse data, it may suffer from a serious simulation-to-reality gap that causes the solution to fail to generalize to the real world.
Engineering your AI success
Companies worldwide are already spending significant amounts of resources on AI/ML programs because they see them as a critical piece of the organization’s long-term success. To make sure those resources are well spent, the best approach is to start small, but without losing sight of what will be required to develop and grow beyond a pilot project. Focusing on smaller use cases with a clear ROI — while using real data and making sure enough organizational muscle exists to ramp up and scale a proof-of-concept into a deployable solution — will ultimately reveal opportunities to make your AI programs more effective and profitable.
