6 Tips for Making “Wicked Hard Decisions”
I was VP of Product at Netflix. Here’s what I learned about decision-making there.
Can high stakes decision-making be learned, or is it strictly the domain of “strategic geniuses”? And if it can be learned, what are the principles that enable you to confidently make tough decisions without nagging fear of failure?
For the last two years, I’ve given a talk called “Wicked Hard Decisions” which has five “What would you do?” cases that provide an insider’s view of how Netflix made tough product and business decisions. I learned a lot while at Netflix and even more teaching these cases.
The skills can be learned. Below I list six principles that will improve your odds of making great decisions. I also outline how you can practice decision-making everyday, whether you are CEO or have just begun your career as a product manager.
Jeff Bezos gets at the first principle in his 2016 Annual Report:
“Never use a one-size-fits-all decision-making process. Many decisions are reversible, two-way doors. Those decisions can use a light-weight process. For those, so what if you’re wrong?”
One of the first questions to ask: “Is this a high stakes or low stakes decision?” If it’s a high stakes decision — like the launch of streaming in early 2007— you’ll need a deliberate approach by a cross-functional team composed of high judgment individuals. But for low stakes decisions — like killing a feature used by only 2% of our members — a small team, close to both the problem and customers, can make a faster decision. And if they’re wrong, they can reverse or adapt quickly. In looking at the variety of decisions companies make, most are reversible, so bias towards action helps you to execute and learn quickly. Unfortunately, many teams use heavyweight processes for all decisions.
I’m guilty of this, myself. At Netflix, I belabored the decision to remove an early incarnation of sub-accounts (“Profiles”). We eventually killed the feature, but then were overwhelmed by the negative response of the 2% of our members who used the feature. (It turns out that most of our board members were Profile users and felt we were ruining their marriages by forcing them to share queues.) We reversed course and all was well. But I’ll never get back the six months I struggled with the decision. I would have been far better off executing quickly, and then made adjustments based on member response.
I define “consumer science” as the search for consumer insight using a mix of data sources:
- Qualitative. Focus groups, interviews, and usability sessions help to define the problem space, to get inside the heads of customers, and to learn how to package and position ideas in ways that resonate with consumers. But I don’t use these sources to make decisions; a few folks in a focus group does not adequately represent your total audience. Qualitative sources help frame how customers think about the product, let you see the product through a new customer’s eyes, and enable you to generate lots of new ideas.
- Surveys. I use this tool to understand the demographics of customers and to see what new features or attributes they might value. Again, I don’t use this source to predict behavior. What customers say and what they do aren’t always the same.
- Empirical or existing data. Increasingly, companies have tons of data — both financial and behavioral — that needs to be organized in meaningful ways. My focus here is to build a suite of metrics that helps form hypotheses about what drives customer and shareholder value. Later, I test these hypotheses via A/B tests.
- A/B Testing and Machine Learning. These are the most effective tools in the consumer science toolbox. The ability to push out a new feature to a small subset of members in an A/B test helps build a low-risk, “let’s test it!” mentality and predict outcomes when you release the change to all. Today, Netflix’ personalization effort relies on algorithmic innovation that combines offline machine learning experimentation with online A/B testing to better predict which movies members will love — or not.
Today, world-class consumer tech companies like Facebook, Amazon and Netflix are quick to embrace the “let’s test it!” mentality and have made large investments in both A/B testing and machine learning. This makes decision-making straightforward. These companies already have a high confidence prediction about the expected result when they push large-scale change to customers.
Netflix’s recent decision to switch from a five-star rating system to a thumbs-up/thumbs-down model was straightforward. They tested the new approach with hundreds of thousands of new members and discovered a 200% increase in ratings using the simpler, “thumbs up” system. Yes, there was grumbling from existing members like me (I hate change!), but Netflix knew via A/B test results that members would not cancel the service because of this change. (I’m still a member!)
Most companies don’t operate at the scale of Netflix, Facebook, Google and Amazon, but that doesn’t mean they can’t develop consumer insight. With fast-growing startups, I have found focus groups outside the Bay Area (away from Silicon Valley freaks like me) to be very helpful. A well-executed Net Promoter Score survey can help startups to evaluate product-market fit. And, a well-developed set of e-staff metrics can help form hypotheses about potential drivers for customer delight.
These four sources of data — qualitative, survey, empirical, and A/B testing — help form a customer-focused mindset that ensures consumers are represented in key decisions. Today, the companies that fully embrace “consumer science” lead the pack in the consumer tech industry.
When I worked at Netflix, I defined my job using a simple model: to delight customers in hard-to-copy, margin-enhancing ways. At the time, I thought the job was to balance customer and shareholder value. By stepping away and thinking about my good and bad decisions, I’ve learned to focus even more on customer delight. To do this, you have to be maniacally focused on delighting your customers.
There’s a project at Netflix I worked on that gnaws at me ten years after the fact. During Netflix’s DVD era, I got excited about a new merchandising algorithm that took into account the cost of each DVD we delivered to customers. At the time, a new release DVD cost three dollars to deliver to a customer, while a DVD that was three to twelve months-old cost two dollars, and a title that was more than one year-old cost only a dollar to deliver. By implementing a cost-based algorithm, we could add millions of dollars to our bottom line. But I got a tepid response from our CEO about the idea. His initial fear was that we would merchandise cheaper titles ahead of titles we knew customers would love. I assured him that we would only “nudge” a lower-cost title when the customer’s predicted rating for each movie was the same. We executed the algorithm and lowered content costs by millions of dollars — it felt like a lot of money at the time.
As I think back on the project and Reed’s lukewarm response and then spend time watching companies like Amazon obsess over its customers, I’ve learned to focus even more on customer delight. I think Reed’s ambivalence reflected his worry that I was too focused on building shareholder value and not enough on delighting customers. In hindsight, I wish I had focused nearly exclusively on the projects that would potentially deliver delight — bigger catalog, streaming on all devices, better video quality, voice control, interactive storytelling, and any number of things Netflix is likely testing now.
My guess is that one of Netflix’ most disastrous decisions — the decision to separate the streaming and DVD businesses via the launch of “Qwikster” — was due in part to the failure of Netflix to obsess over its customers. Splitting the service in two required that customers create and manage two separate accounts, resulting in customer outrage — the opposite of delight. (Yes, there was a price increase, too, which was hard for customers to stomach, especially as both the price increase and Qwikster changes were poorly communicated.)
Today, Jeff Bezos describes the benefits of customer obsession in a way I more fully appreciate:
“There are many advantages to a customer-centric approach, but here’s the big one: customers are always beautifully, wonderfully dissatisfied, even when they report being happy and business is great. Even when they don’t yet know it, customers want something better, and your desire to delight customers will drive you to invent on their behalf.”
As you work to satisfy your inherently unsatisfiable customers and contemplate the effect your decisions will have on them, you will find yourself relentlessly driven down the path of innovation.
Do you have a well-articulated company and product strategy? If you do, you’re in a good place to make great decisions. I view product strategies as high-level hypotheses about what delights customers in hard-to-copy, margin-enhancing ways. In making decisions ask, “Is this decision on strategy?” If the answer is no, don’t do it.
In 2005, Netflix added a shipping hub in Hawaii and retention improved as disks took one day instead of three to arrive at a member’s mailbox. This retention improvement led to the hypothesis that faster delivery of DVDs nationwide would substantially improve retention. This strategy — deliver movies “instantly” — guided lots of projects to deliver disks faster to customers.
We measured our progress through a metric that described the percent of times members got their first choice disk the next day in the mail. Our “instant” strategy, with measured progress against the metric, fueled lots of projects — doubling the number of shipping hubs in the US, tuning our merchandising algorithms so that only the disks in your local shipping hub were displayed on the site, and eventually led to the launch of streaming. Most of these decisions were straightforward as they were very much on strategy and we could measure progress via a precise metric. This metric improved from 70 to 95 percent over the course of a few years and I’m confident it led to retention gains — Netflix’s ultimate measure of delight.
Most decisions are low stakes and reversible, so your bias should be to make them quickly. But what about the high stakes decisions? What’s the right pace for these decisions?
When I was a college student, I got my pilot’s license. A lot of the flight safety training focused on high-risk, fast-paced, decision-making. You are trained to make decisions quickly, knowing you can adjust after the fact. You also learn that doing nothing — when flying into a cloud without an instrument rating, for instance — is rarely the right decision.
In flying, you’re introduced to the concept of provisional decision-making. The concept: “With the data you have right now, what decision would you make?” Then ask yourself, “What additional data do I need to verify this decision? How and when can I get that data?” In the instance of flying into a cloud, the common provisional decision is to make a 180-degree turn. The additional data required is more weather information about clouds at different altitudes and potential alternate routes.
In 2007, Blockbuster engaged in an assault on our business with their “Total Access” program. They offered their DVD-by-mail customers unlimited disk swaps at their stores. We debated an appropriate response — lower prices or do nothing — but the additional piece of data we needed was, “How many disks are Blockbuster customers watching each month, and based on this, how long can they afford to continue the program?” We gave ourselves a month to get this data and found it. The answer: Given a high rate of in-store “disk swaps” by their customers, Blockbuster could only afford to maintain the Total Access program for nine months. Our response: double down on streaming. By the time Blockbuster stopped “Total Access” (at nine months!), we had a substantially better streaming service, while Blockbuster had none.
A simple rule of thumb in decision-making: Get 70% of the data you need, then make a decision. If you have less data than this, you’re potentially reckless. If you have more data than this, you’re probably wasting too much time.
Growing up, my parents would say to me, “Good fights make good marriages.” (They’ve been married 55 years!) This folksy, New England saying reinforces the idea that you need good debates to make great decisions together. And once the decision is made, you need to quickly align to execute. Amazon calls this behavior, “disagree and commit.”
At Netflix we worked to deliver overwhelming value to our customers. Price was a big lever. We A/B tested pricing but there were many debates as we interpreted the results: “How to balance Wall Street’s expectation for profit against the growth that lower prices delivered? How much debt should we take on in order to fund future growth?” There were many other, potential long-term impacts of price changes that we couldn’t evaluate via A/B tests.
Every Monday morning Netflix had an executive team meeting and I can still remember Reed, Barry McCarthy (CFO) and Leslie Kilgore (CMO) modeling the behavior they expected of the broader executive team. Results of A/B price tests in hand, they would debate the more nuanced issues of pricing. And from time to time, Reed would ask participants to flip their point of view and argue the opposite — an exercise that encourages not just debate, but careful listening.
A common question I get: “You had passionate debates at Netflix, but how did you ensure everyone got on board with the decision?” My response, “It was part of the culture.” To this day, I can still remember phrases that were repeated to encourage the behavior: “disagree and commit,” “decide and do,” and the simple concept of “well-formed adults” who could switch seamlessly from candid, passionate debate to lockstep execution as a team.
Making great decisions — about product, people, and business — requires practice. But how can you practice decision-making in your everyday job?
It’s actually quite easy. In any meeting where a decision needs to be made, ask yourself, “What questions do I need to answer to make this decision?’ then ask the questions! Then, ask yourself, “What should we do?” regardless of whether you are the decision-maker. Your final step: bravely state your opinion and work to initiate meaningful debate.
Over time, you’ll develop the necessary decision-making skills and people will seek out your opinion. One day, you’ll find your CEO asking, “What would you do?” You’ll know at this point that you are on the way to making your own “wicked hard decisions.”
If you enjoyed this post, I did a follow-up piece called “Four More Decision-Making Principles” that you can read by clicking here.
I’d love to hear your feedback on this post, as well as your experiences in making tough product and business decisions. And if you’d like to learn more about my “Wicked Hard Decisions” talk, click here.
Thanks!
Gib
Gibson Biddle
My Other Product Management Articles on Medium:
- Netflix’s Customer Obsession
- How to Run a Quarterly Product Strategy Meeting
- Branding for Builders
- Hacking Your Product Management Career
- Leaders Lead
- How to Find a Great Job
- What Your Next Head of Product Looks Like
(Many thanks to Florian Fischetti, Ashita Achuthan and John McMahon for their edits, and for John Russell for graphic help.)
Click below to:
Enjoyed this article? For more on entrepreneurship, start-ups and product leadership, get our monthly Insights newsletter delivered to your inbox. Subscribe here.
