Improving products with the wisdom of crowds

(and why you can’t outsource innovation)

Be careful about the customer

An interesting question any product person faces is “when should I rely on my customers for product feedback, and when should I not?”

After all, “talk to your users” is so often repeated it’s now a truism. But when, if ever, should we not rely on client feedback?

I believe the answer is straightforward and invaluable: you can (and should) use customer feedback to iterate (defined here as making an existing product better), but you cannot rely on clients to innovate (defined here as making a new product).

The best summary of this concept is found in this interview with legendary “Coach” Bill Campbell, the former CEO of Intuit and trusted advisor of Apple, Google, and Amazon:

Bill has it exactly right: clients can make your existing products better. But they can’t conceive of new products. And any leader who relies on user feedback when building something new is destined to fail.


Innovation — Or, making new products

“It’s really hard to design products by focus groups. A lot of times, people don’t know what they want until you show it to them.” 
-Steve Jobs

Figuring out the right product is the innovator’s job, not the customer’s job. Customers typically imagine in things that already exist, and if they knew what was next, they’d be building it.

It’s common to mistake “customer development” for “if I just talk to enough users they will tell me what to build.”

No.

Customer development is to test your hypothesis, your unique insight, not to generate insight. Start with your unique insight; your vision of the future state of the world.

When products are built by a collection of user requests — focus groups — they are, by definition, consensus products. And consensus products will not be breakout successes.

Apple is the best example of the ignore-the-customer-when-innovating approach, starting with the touchscreen back in 2007 all the way to the killing of the headphone jack, the home button, and cordless headphones in 2018.

My favorite story on this comes from Marc Andreessen:

In 2007, right before the first iPhone launched, I asked Steve Jobs the obvious question: The design of the iPhone was based on discarding every physical interface element except for a touchscreen. Would users be willing to give up the then-dominant physical keypads for a soft keyboard?
His answer was brusque: “They’ll learn.”
Steve turned out to be right. Today, touchscreens are ubiquitous and seem normal, and other interfaces are emerging. An entire generation is now coming of age with a completely different tactile relationship to information, validating all over again Marshall McLuhan’s observation that “the medium is the message”.

or consider this quote from Jeff Bezos:

Market research and customer surveys can become proxies for customers — something that’s especially dangerous when you’re inventing and designing products. “Fifty-five percent of beta testers report being satisfied with this feature. That is up from 47% in the first survey.” That’s hard to interpret and could unintentionally mislead.
Even when they don’t yet know it, customers want something better, and your desire to delight customers will drive you to invent on their behalf.

Iteration — Or, improving existing products

So, when does it make sense to tap into your customers for insight?

I used to watch “Who wants to be a millionaire” growing up, and I was fascinated by how often the audience was correct when they were polled.

Especially on the really tricky questions :)

As it turns out, the audience was right 91% of the time, and this was a prime example of the “wisdom of crowds” phenomenon.

When a group of people is independent, decentralized, diverse, and their opinions are aggregated, the wisdom of crowds becomes a source of insight. And these criteria typically hold if you’re looking at existing software products. Given a constrained problem (how do I make this better), with a fairly clear solution set, the wisdom of crowds becomes an incredibly powerful tool to find the optimal result.

For example, the product I manage at Wealthfront is our advice engine, Path. It was originally built to help our 250,000 clients plan for future financial goals like retirement. And when we first shipped it in January 2017, retirement was the only “goal” we supported. But we quickly learned that our clients had a bunch of other large financial goals, and we had to prioritize the next one to build. Client feedback proved valuable here, and we quickly learned the next problem they had in their financial life was buying a home. So we used client feedback to “iterate” on Path, by prioritizing a new home planning experience.

But, what exactly should the home planning experience be?

Here’s where it gets nuanced: the “home goal” was a new product in itself, and we knew we’d need to innovate to make it work. There was no other holistic home planning experience available anywhere else (just simple home affordability calculators), so we were back in innovate mode.

Using a combination of customer development, user testing, and product intuition, our team built a delightful home planning experience (not relying on focus groups or feature surveys) and added it to Path.

And — once it shipped, we were back to iterate mode, talking to hundreds of our users about how to make home planning (a now-existing product) better.

Since then, we’ve continued to improve Path by allowing users to plan to plan for a child’s college or to take extended time off to travel, making Path the only holistic, free financial planning experience available anywhere. And all along, we used both the wisdom of client feedback to iterate and improve Path, as well as the tricky innovation process to create the new product features.

I learned that great product teams jump seamlessly and often between innovation and iteration.

After all, even the most innovative visionaries use client feedback to iterate to the top of their existing product mountain:

Building a faster horse

The cliche quote on this concept is (supposedly) from Henry Ford:

“If I had asked people what they wanted, they would have said faster horses.”

And of course the point is correct: if Ford, Steve Jobs, and other visionaries relied on customers to innovate, we’d all be far worse off. Great products are born from great insight, and great insight comes from great product minds.

But, this conclusion neglects one obvious point: often, product leaders need to figure out how to iterate on the horse. And the customers in the saddle know how.


**Thank you to Andy Rachleff who not only inspired the ideas behind this post but reviewed it and provided feedback**