“The Ladder of Evidence” for AI

I really liked this talk by Teresa Torres on “The Ladder of Evidence.” I suggest you watch it (or check out the post) if you have anything to do with product research:

She is absolutely right there is a tradeoff between time/effort and value. I have been thinking a lot about the use of intelligent algorithms and it resonated with my previous post “Testing AI concepts in user research.”

How do you apply this ladder to user research for AI, deep learning, and machine learning?

First, problems first

Before we move forward, I want to make a point that what is most important is to solve real world problems when integrating intelligent algorithms.

I can’t repeat this enough. You won’t get the ROI on AI unless you focus on that first.

“Ask what they would do” vs “Ask what they’ve done in the past”

As a quick aside that isn’t related to AI/DL/ML specifically, at the bottom of the evidence ladder the value is low in comparison to the effort. Simply changing the way you interview to be less leading and more focused on past experiences isn’t more time or effort. It is just experience and training.

Teresa has another great post on Product Talk about “Why you are asking the wrong customer interview questions” on this topic.

“Observe them during a real life experience”

We advocate the high value observations of people (á la contextual inquiry) to really understand what they are doing and how they are cobbling solutions together.

In the case of intelligent algorithms you should be trying to observe any of the following:

  • Manuals the people are referencing
  • Lists that they have on their desk to look something up
  • People they need to call for information
  • Hard decisions they are making

All of these observations are interesting when trying to understand how an intelligent algorithm could help out. Let’s look at this list again with possible ways intelligent algorithms could help:

  • Manuals they are referencing — recommend next steps as understood by the intelligent algorithm.
  • Lists that they have on their desk to look something up — match data that isn’t easily matched using intelligent algorithms.
  • People they need to call for information— generate possible answers that a person would offer or automatically add the right person to the workflow.
  • Hard decisions they are making — these are great opportunities to show possible forecasts of the future if they take different actions.

This research is a great precursor to building a prototype…

“Simulate a new experience”

Even though it is easier than ever to build ‘live code’ to try out during testing we don’t have that luxury when dealing with intelligent algorithms. Due to technology stage intelligent algorithms are at, there is still a large cost to building out a possible concept in an experiment.

Since prototyping is really time/effort-cheap today it is a great when considering what types of intelligent algorithms are helpful. For more on prototyping check out the “Testing AI concepts in user research” post:

At Philosophie the designers Leah and Jamie would put together screens in Sketch and wire them up for interaction in InVision. We don’t assume that the person can do everything with the prototype. If they try to do something that isn’t covered we simply instruct them it isn’t finished yet.

The prompt we would use would generally be something like “You get into the office in the morning and you want to start taking care of incoming jobs.” From there they have the option in the prototype to click on an intelligent call to action (how the intelligent algorithm helps) to schedule a job.

Intelligent CTA is a very simple, but powerful, concept

How intelligent CTAs work in the intelligent algorithms context will be in a future post. Stay tuned!

To simulate different ways the intelligent algorithm would help we would have slightly different circumstances that include special cases that the intelligent algorithm is wrong (e.g “the person recommended isn’t the right technician for the job, what do you do?”). It is interesting to see how the person’s trust changes over time.

Key take aways

The ladder of evidence is a great way to think about the cost vs. benefit of research and I’m really happy that Teresa Torres wrote it. The benefit outpaces the cost as you move up that ladder.

If you are thinking about integrating intelligent algorithms into your products make sure what you are building is worthwhile first. We need to move from the ‘toy’ mentality I see in a lot of AI/DL/ML demos and into solving real world problems.


Philosophie is a software design and development consultancy located in Los Angeles, New York, and San Francisco. We unlock innovation by eliminating the strategy-execution gap. Let’s get to work and make something that matters.