Performance management for AI
Turo, from QuantumBlack, AI by McKinsey
This article examines how QB Labs developed Turo, which measures the impact of AI projects in terms of value tracking and performance management. Turo offers businesses a holistic view of their AI projects by integrating the output of their analytics tools. It enables visibility on individual use cases, and increases productivity by identifying issues and ownership across AI projects within the business. Clients across a range of industries have reported a 25–30 percent increase in productivity through the automation of repetitive key performance indicator (KPI) tracking and reporting activities. This equates to saving an entire week per month for data scientists or engineers that would otherwise manually collect data, integrate it into reports, and share it with stakeholders.
QuantumBlack Labs is the R&D and software development hub within QuantumBlack, AI by McKinsey. QuantumBlack Labs has more than 250 technologists dedicated to driving AI innovation and supporting and accelerating the work of its more than 1400 data scientists across over 100 locations. We use our colleagues’ collective experience to develop suites of tools and assets that ensure AI/ML models reach production and achieve sustained impact.
Projects using artificial intelligence and machine learning (AI/ML) are a key driver of business value, and companies have shifted from exploring what the technology can do to exploiting it at scale to gain market share. Companies that successfully capture the full economic potential of digital innovations are those that rigorously track the value and performance of each advanced analytics initiative.
Impact tracking for scale
Technical teams need to show that investments in AI capabilities and use cases are sustainable and justifiable ahead of significant scaling or further project expansion.
As chapter 30 of our book, Rewired: The McKinsey Guide to Outcompeting in the Age of Digital and AI (Wiley, June 2023) describes, effective impact tracking can justify future investment and transform AI into a tangible business metric.
For investors to make optimally informed decisions, there must be clear visibility on AI/ML projects. Strategic value-tracking helps them prioritize the most impactful AI initiatives and determine the data assets and technical infrastructure needed to support them. Some of the measures include:
- Leading indicators: These predict future events or trends and are used to anticipate changes and provide early signals of what might happen in the near future
- Lagging indicators: These reflect past events or outcomes and are used to confirm trends and provide feedback on what has already happened.
- Investment: How much is the business investing to build the AI portfolio?
- Third-party costs and data: How much are we spending on third-party tools and API access? How much are we spending on data sets and gaining access to data?
- Adoption: Are we getting the right level of adoption from the business for each use case? What is the baseline for future solutions?
However, the complexity of AI/ML projects can make it hard to measure impact and represent it effectively, particularly in larger companies, for reasons such as the following:
- Different teams may use multiple disjointed tools to manage AI use cases.
- Stakeholders find it hard to monitor their use cases, particularly when they become numerous, and are in different phases of rollout.
- Use cases have different business, security, and legal compliance aspects, and there needs to be a way to indicate which factors apply to which use cases.
- There is no central way to track and compare the performance, costs, productivity metrics, and adoption rate of analytics use cases.
Turo by QuantumBlack Labs
Turo offers performance management for AI, both for generative AI and traditional machine learning projects. Accessible via a web application, Turo provides fast and simple insights-reporting across the entire AI portfolio, such as:
- Deep dives on individual use cases
- Leading indicators and KPIs
- Issue identification, such as compliance issues, budgetary caps reached or new opportunities.
- Use case ownership and access management.
Key features of Turo include:
Portfolio overview — Seamlessly gathers data from diverse systems to provide a unified view on all the AI projects within an organization. Offers insights on project progress and maturity, and identifies potential bottlenecks.
Compliance tracking — For end-to-end management and mitigation of AI and gen AI risks and compliance with AI regulations and internal policies.
Metrics reporting — Ensures AI projects align with the business goals by tracking business and performance metrics, with alerts whenever values fluctuate.
Cost tracking — Oversight on how much each request, session, or cloud instance costs by consolidating data on infrastructure or LLM usage into a simple, unified visualization to prevent overspending and unnecessary expenses.
Turobot — For quick insights, proactive recommendations, or detailed project cost reports, the Turobot assistant enhances decision making.
Impact of Turo
Clients across a range of industries have reported a 25–30 percent increase in productivity through the automation of repetitive key performance indicator tracking and reporting activities. This equates to saving an entire week per month for data scientists or engineers that would otherwise manually collect data, integrate it into reports, and share it with stakeholders.
Leaders have also noted that Turo has significantly improved their ability to communicate value upstream in a more intuitive and interactive manner, thereby increasing confidence in the AI function across various parts of the business.
As one business stakeholder puts it, “Having all AI projects on Turo helps us identify existing AI capabilities that have already been developed or purchased, thereby avoiding redundant efforts”.
Does my team need Turo?
The following are all clear signals:
- There is no single inventory of all use cases within the organization.
- Investors cannot track spending and return on investment for any particular use case.
- Stakeholders are using basic spreadsheets to track how an AI portfolio is performing, and have no access to live dashboards of realtime data.
- Compliance against internal and external policies is complex and cannot be connected back to the impact-tracking elements of a use case.
Summary
Strategically, organizations need an inventory of existing data and AI products. The optimum format is a set of visual dashboards that monitor the performance of every analytics use case to facilitate decision making.
Turo brings a holistic view of projects across business units and regions by integrating with the main analytics tools the organization uses. It enables swift insight generation for stakeholders who need to understand the impact and status of their AI use case portfolio.
At QuantumBlack, AI by McKinsey, we have seen first-hand that moving AI solutions from idea to implementation can be challenging. We recognize that scaling an AI/ML project relies on measuring what matters and tracking its impact so that the business can assess the value of every use case delivered.
QuantumBlack Horizon is a family of enterprise AI products, including Kedro, Brix, and Alloy, that provides the foundations for organization-level AI adoption by addressing pain points like scaling. It’s a first-of-its-kind product suite that helps McKinsey clients discover, assemble, tailor, and orchestrate AI projects.
To learn more about what QuantumBlack Horizon can do for you, please email Yetunde Dada.
Thanks to all who contributed to this article: Mohammed ElNabawy, Fabiana Ferrara, James Mulligan, Jo Stichbury, Joanna Sych, Sarah Mulligan & Matt Fitzpatrick.
