How to avoid the 85% of data science project failures
Why it is essential for us to transform into a Data Strategist ( ~ Data Science + Decision Intelligence ) as 60% of the data science projects would never go into production as per Gartner analysis and that number may spike up to 85% if we are less conservative about the estimate.
How can we avoid/overcome these pitfalls?
Few years ago, data science was often termed as the hottest job in the market and business was behind these data masters to extract the state of the art insights/analysis to drive company. Most of these analysis did not meet the expectations due to four major factors where things went wrong.
- Too broad of an outcome
- Business Impact
- Trying to repeat the data outcomes
- Inclusion of data science during the end of data journey
1. Too broad of an outcome
One of the most common issues/pitfalls in data science has been making a broad decision and develop steps to achieve that. For Example, increasing sales and increasing the revenue are too broad and it fails narrow down the path and define a metric.
There needs to be a metric or an business impact which we are planning to resolve which can also be captured and measured to see the outcome.
For Example, improving sales and revenue is too broad of an outcome compared to improve the renewal rates in enterprise customers for software products by x%. We have a clear baseline/lift here to achieve than improving overall numbers with random patterns.
2. Business Impact First
There are only 2 constants in the industry. One is Business value and second is the speed we realize/execute that value.
Our analysis should always be started in a way to see the impact going to be created by having a frequent brainstorming sessions with the business leaders and also to be able to quantify the outcome. There needs to be a value associated with the AI/ML project which will create an impact and the uplifts we can get from these models.
3. Trying to repeat the data outcomes
We often tend to mimic the data outcomes based upon the earlier projects and it may lead us to the loop of same business outcome. To avoid this we often need to use our knowledge as launchpad to quick start rather than mimicking the outcomes of older models.
We have already seen the impact of disruption by the pandemic which has disrupted all our forecasting/predictions as the historical numbers which was stored since ages is less of value if business/AI is not adapted to the change in data patterns today. In short we should have a plan to rebuild the models/analysis which change in time rather than recreating same model and expect everything to work as earlier.
4. Inclusion of data science during the end of data journey
We have often seen data science teams being pitched in during the last phase of the projects after data strategy, governance and building the infrastructure setup for day to day data operations.
Data science has to be part of an organization’s data DNA. Rather than hiring it last, it should be part of the data governance process.
Bryce Macher Senior lead of product analytics, CNN
By inducing the data scientists during the data strategy, we give the super access and inform the business impact or outcome with much cleaner data points to begin with.
Also, read through the other stories which I have published on medium if you are interested in learning sales and marketing use cases in data science.
Thanks for reading!…
If you thought this was interesting, leave a clap or two, subscribe or watch out my LinkedIn posts for future updates!!!
Added to publication Data Dreamers.
References:
