My team at Kromatic and I originally published this post here on Grasshopper Herder, our lean startup blog. Kromatic’s mission is to help companies get to market fast and reduce the cost of innovation. Subscribe to our newsletter or speak to one of our coaches to see if we can help you reach your innovation goals.
There are two lean startup methods that are often confused with one another. As a result, they’re often misused: Concierge test and Wizard of Oz test.
tl;dr: Wizard of Oz is for testing a specific solution hypothesis, Concierge is for generating ideas. Use each for the right reasons. You can read more by downloading the Real Startup Book.
Wizard of Oz Test
In the lean startup world, the Wizard of Oz method was used by companies like Aardvark and CardMunch. Both were able to create amazing effective prototypes very quickly by not building anything.
Aardvark connected people with questions to people with expertise. Cardmunch managed to transcribe blurry photos of business cards better than any other Optical Character Recognition (OCR) system at the time. Neither had any technology or algorithm behind them.
The technology was human beings.
Cardmunch leveraged Amazon’s Mechanical Turk while Aardvark used interns. Instead of building complex algorithms both just faked it. Since neither needed a real time response to provide their core value propositions, there was no need to build anything.
To the user, the products delivered exactly what they promised. A little slow perhaps…but the value was there.
(Note: Lean Startup did not invent this technique. It was already being used in the lab by 1975 by J. F. Kelley.)
Concierge Test
Concierge testing was used by companies such as Wealthfront and Food on the Table. The approach is similar in that there is no technology involved. Both companies performed the service they were offering as a concierge at a hotel might. Purely manually.
Wealthfront would work with its customers to provide wealth management advice and portfolio management. They would sit with them, pen and paper in hand, and figure out solutions to wealth management issues step by step.
Manuel Russo used to personally go shopping with his customers to help them plan their meals. (Not very scalable!)
So both Wizard of Oz and Concierge testing provide the value proposition manually without any technology. What’s the difference?
User Experience
The difference is that in one case, the user knows about the concierge, while the Wizard of Oz is hidden from the user. This is a key difference for the user experience.
In a Wizard of Oz test, the user experience is a complete simulation of the actual intended product, but with a slight time delay since a human cannot react as fast as a computer. The delivered value proposition is almost identical but slightly under performing.
In a Concierge test, the value proposition is actually significantly higher than the eventual product. That’s because the customer is aware and is interacting with a real human being.
Humans are Social Creatures
Human interaction, as a whole, is generally a positive value proposition. We’re social creatures and even introverts gather in groups to socialize.
There are some circumstances where we do not want to interact with a human being. For example, we don’t want to talk to anyone at Macy’s. Malls and department stores just kind of freak us out in general.
For everyone else, just imagine that you just typed a search into WebMD to do a little private research and all of a sudden a window pops up with a live human being on a webcam to tell you all about your search query for genital herpes. Fun!
Compare that to Food on the Table where we’d get to go shopping with Manuel Russo. Normally, we find shopping a bit of a chore, but Manuel is a pretty fun guy. We were delighted to meet him a few years ago in Austin and would be equally delighted to go shopping with him. (Manuel seems to know all the best places to get food in the city and is hysterically funny.)
A Concierge test has an extra value proposition that a Wizard of Oz test does not. Or at the very least, the test is severely affected by the visible presence of a human being.
Generative vs. Evaluative
This brings us back to the distinction of Generative Research vs. Evaluative Experiments.
Wizard of Oz testing allows us to test a clearly defined hypothesis about what solution will provide the value proposition to the customer. We test it by simulating the user experience as precisely as possible without technology. The lack of technology means we can rapidly test a variety of hypotheses and rapidly iterate.
Since Wizard of Oz allows us to test and falsify a solution hypothesis, it’s an Evaluative Product Experiment.
Concierge testing has a huge human factor which biases the test. If we attempt to use a Concierge test to validate our solution hypothesis, we’re liable to get a bias towards a false positive or perhaps even to a false negative in some circumstances. It’s simply the wrong type of test to falsify a clear hypothesis.
Concierge testing is very bad when we have a clear idea of the solution, but very good when we’re not exactly sure what the solution should be!
Generating Ideas
When Wealthfront sat down with their customers, they had some idea on how to manage finances, but they didn’t have a step 1, 2, 3 algorithm of how to do it.
There were many unknowns including how to best represent the portfolio information to the customer in an intelligible way to them to make their own decisions.
Concierge testing allows us to work with the customer without having a clear solution hypothesis. A vague idea, or even no idea, will sometimes be enough to muddle through the process while talking with the customer and getting constant feedback.
We might think that it’s impossible to go to a customer and just sell the promise of a solution without having any clear idea of how to provide it. It’s not.
Selling solutions that don’t exist and then winging it is common place. It’s called Consulting. -Tweet This
(Note: Lean startup didn’t come up with Concierge testing either. Consulting has been around since the before the pharaohs.)
By the time we’ve manually performed the task a few times for our consulting clients, we have a much clearer idea of what form an automated solution can take and what the minimum viable feature set looks like.
The solution is not yet validated, but we can take that hypothesis and begin the process of evaluating experimentation with a Wizard of Oz test, paper prototyping, usability tests, etc.
Lessons Learned
Wizard of Oz hides the human. Concierge flaunts the human. -Tweet This
The Wizard of Oz method is for evaluating a solution hypothesis. -Tweet This
The Concierge method is for generating solutions ideas and hypotheses. -Tweet This
Confusing them can lead to faulty reasoning and flawed business models.
For more ways to test your solution or figure out your minimum viable feature set, download the Real Startup Book which is being co-created by 40+ lean startup enthusiasts:
My team at Kromatic and I originally published this post here on Grasshopper Herder, our lean startup blog. Kromatic’s mission is to help companies get to market fast and reduce the cost of innovation. Subscribe to our newsletter or speak to one of our coaches to see if we can help you reach your innovation goals.
