CEO to CIO: “What’s our AI strategy?” — Part 2

By Robin Bordoli

This post is the second in a three part series. In the first post last week, we gave some insight and context into why your CEO is asking this question, why now, and why you. In this second post, we will give you a foundational framework to think about AI so you can give your CEO a thoughtful response.

CIO: “How do I respond?”

Last week we introduced the metaphor of the CIO being able to ride the waves of technology disruption. To avoid being knocked off your board and swept out to sea, first you must develop a solid conceptual framework for understanding AI. Once you have that, you can engage with the business, and evaluate your options for working with vendors providing AI solutions.

Formulating a Framework for AI

Let’s start off with some definitions of what AI is not. This is necessary because the media coverage of AI is not thoughtful but often hyperbolic.

Let’s tackle Myth #1.

AI ≠ Machines > Humans

For the last 30 years the media has loved to portray AI as the replacement of humans by machines, whether it’s Schwarzenegger in the Terminator or Alicia Vikander in Ex Machina. This is the wrong mental model for AI in the enterprise. The right framing is how can machines augment humans, not replace them. Even the recent the media coverage of Google’s DeepMind/AlphaGo victory over Lee Sedol was simplistically portrayed as Machine defeats Human. The more accurate description would be Machine plus many Humans defeats single Human.

Machines have advantages that Humans do not: speed, cost, consistency. Humans have advantages that Machines do not: task complexity and breadth of task capability. The challenge is to find the right way to blend Humans and Machines, not replace Humans with Machines. As a reminder that Machines aren’t ready to take over from Humans just yet, a number of robots in DARPA’s robotics challenge last year struggled to open a door.

Now let’s tackle Myth #2.

AI ≠ Best Algorithm

For many people the terms AI and algorithm are synonymous. The best algorithm creates the best AI solution. Facebook has the best newsfeed algorithm, Netflix has the best movie recommendation algorithm, and Google has the best ad placement algorithm.

We think this is incomplete. AI and algorithm are not synonymous terms. Algorithms are a necessary component of AI, but not a defining one. Many leading experts such as Alexander Wissner-Gross now claim data — and not algorithms — are the key limiting factor to development of human-level artificial intelligence. He reviewed the timing of the most publicized AI advances over the past 30 years, with the evidence suggesting many major AI breakthroughs have actually been constrained by the availability of high-quality training datasets.

To further illustrate this, let’s take a look at some of the research published by Google on artificial neural networks (ANN) that use learning algorithms. Google was training an artificial neural network — a computational model that draws inspiration from how brains work — in image classification. One of the tasks was identifying dumbbells. The ANN was trained by showing them it many examples of what Google wanted them to learn, hoping they extract the essence of the matter at hand (e.g., a dumbbell needs a handle and 2 weights), and learn to ignore what doesn’t matter (a dumbbell can be different weights, sizes, colors, or orientation). Unfortunately the ANN failed to completely distill the essence of a dumbbell.

Maybe it’s never been shown a dumbbell without an arm holding it. These mistakes highlight that we need to be careful before completely removing humans from the process. A human can immediately see the mistake while a machine cannot. This element is called Human-in-the-loop (HITL).

So this brings us to a more complete working definition of AI for the enterprise.


We believe this is the essential equation that the CIO needs to understand if AI is to be a commercial success inside an enterprise. So let’s break it down and imagine a company is trying to create an AI solution that can categorize customer support tickets by severity level based on the unstructured text showing an exchange between a customer and a customer support rep discussing a particular topic or problem within the support ticket.

TD is Training Data. Training Data is a set of inputs with the correct outputs or examples with the correct labels that can be used as example to train the Machine. In this example the input is the unstructured text inside a support ticket. The output or answer is the label “severity level” which has been applied by Humans according to definitions of severity levels specific to the company in question. An automotive manufacturer will want to define these severity levels differently from a retail banker or a wearable technology company.

ML is Machine Learning. The Machine Learning capability is the ability to convert Training Data into a predictive model that can be applied to new inputs — in this case new support tickets with unstructured text. You want the Machine Learning model to apply its predictive power to create new outputs — in this case the “severity level” label. One of the advantages of Machines compared to Humans is their ability to understand their own confidence level. Humans are notoriously overconfident at evaluating their own judgments. So you can accept or reject the prediction based on the Machine’s own assessment of its confidence level. For example, if a support ticket has words and phrases which haven’t been seen in the training data, or seen very infrequently, then the Machine will objectively assess its own confidence level as being low for that particular prediction.

HITL is Human-in-the-Loop. This is the critical third component of commercially viable AI. If the Machine Learning model is not confident in its prediction it can route it to humans to review and answer. In this blended model you take advantage of the speed and scale of Machine Learning to address the less difficult tasks, while the humans handle the harder tasks.

Next week in the third and final post in this series, we will talk about how to engage the business and think through important considerations for evaluating vendors.

Robin Bordoli

Robin is Chief Executive Officer at CrowdFlower having joined the company in February 2015. Robin has spent the past two decades helping high growth companies launch and scale platforms and products into rapidly transforming markets. Outside of work Robin spends his time trying to keep up with his two young children and enjoying all that the Bay Area has to offer.