According to 2022 global research from Accenture, while the majority of organizations that use artificial intelligence (AI) are still experimenting with the technology, only 12% are using it at an AI maturity level that achieves a strong competitive advantage.
In order for our products to deliver value for our customers and businesses, and in turn help our organisations acquire a competitive advantage in the marketplace; we need to ensure that we define the business goals and success metrics accurately. Then these metrics can be monitored continuously so that we can course-correct them throughout the AI development cycle.
What are your business goals?
Different organisations can have different business goals. Some of the common ones include:
- improving customer experience
- revenue gain
- improving your user engagement
- reducing the operational costs
- better and faster decision making
Once you have your business goal defined, it is time to refine the success metric. The success metric for your AI product must be aligned with one or more of the business goals identified in your overall strategy.
What are your success metrics?
Success metrics of your AI product should be:
- Easily measurable
- Directly correlated to business performance
- Isolated to, or traceable to, factors controlled by the AI model being deployed
- Comparable to industry benchmarks
Some examples of success metrics include:
- Conversion rate optimization
- Net promoter score
- Customer lifetime value (CLV)
- Customer churn rate
- Return on investment
Outcome vs Output
AI Products must be deployed to deliver specific and measurable business outcomes. However, the machine learning models and algorithms give us definitive outputs and have attributes like accuracy, execution time, recall and precision (we will delve deep into these terms in a future post on training the model)
We need to monitor both output (from the model) and outcome (for the business).
We need to keep monitoring the outputs from the model to determine how accurate it is, the execution time of the model, etc.; but we should always keep an eye on the business outcomes too because ultimately that is what we want to drive.
It is the balance between model accuracy and business outcome which will ultimately tell us when we should stop optimising our model. We may not want to consider a model with the highest accuracy, which is not delivering the right business outcome.
Customer Service Emails example:
Let’s continue with our example of automation of customer service informational queries from the previous posts, and define our success metrics.
In the below figure, on the left side, you have the number of metrics that you keep measuring and improving, for example, the number of customer service emails answered through automation, execution time, number of customer service emails which required manual intervention, which are all relevant to measure the performance of our model.
And on the right side, you are capturing the associated business metrics like customer lifetime value, CCA, and number of new customers which will drive our business goals of revenue gain and improved customer experience.
If you look at the user adoption funnel, these metrics can sit at different stages of the funnel, impacting the whole business and user journey along the way. The number of new wholesale or international customers will be relevant to the conversion stage, while Customer LTV, CCA and retention rate will apply to the revenue stage and customer feedback quality score and customer referral rate is more important for the referral stages. This goes on to show how one AI customer service request automation model can have an impact on various stages of the user adoption funnel and would require metrics to be tracked across these stages. Hence, collecting user data and user feedback across these different stages is paramount.
In Summary:
Collecting the data need for these metrics, benchmarking the metrics against your goal and continuously improving them until the benchmarks are met/business goals are reached will ensure that your AI product is giving you a competitive advantage.
In the next post, we will discuss whether we have the right data for training our machine learning model for our product.
