When Something Works, It’s Clear
Netflix, and being lost in local maximums.
--
When I was 16, a group of friends and I took a summer road trip to Alaska. We piled into an RV and we drove from Salt Lake City, through British Columbia and the Yukon, and into the largest state in the union.
Eventually we found ourselves camping in Mt McKinley National Park (as it was called back then). We were excited to see the largest mountain in the continent, sitting at 20,300 feet. They say that it’s so foggy in the park that the mountain is only visible a few days a week, but ten teenaged boys were going to be patient. As soon as we were set up, the fog seemed to clear and we found ourselves staring at a stunning mountain range across the basin. Idiots that we were, we each were eye-balling the range and pointing out which peak we were certain was Denali, “the great one.” It was sorta hard to tell, but I had convinced myself, after looking at some pictures in a book, that I could identify it. Denali has two peaks, the south peak is slightly higher than the north peak, so there was reason to argue. Over the next days, various peaks seemed to win our affection — was it THAT one? no, it’s THAT one. This went on.
But on our third day in the park we got lucky, more clouds broke, and we realized what we had been staring at were the foothills, some mountains that line up in front of Denali. In fact, behind the range was a single, enormous white mountain. It towered behind them all, more than twice the apparent size, and we were humbled. THAT was the Great One. It was awesome, in the truest sense of that word.
I’ve seen this repeatedly in my experiences: squabbling in uncertainty whether this thing or that thing is the thing. At Netflix, as we did our various experiments in product design and customer research, we’d perform tests. Sometimes the results would come back — was test cell A the winner or test cell B? One edged over the other, but as time marched on, maybe it was hard to tell. For instance, we were exploring lots of “PLAY” button design and placements to encourage more streaming. It seemed like a bigger button was better than a small one, a red one better than a blue one…While we felt we could determine the winner, others would decry these results (with total political inappropriateness) as “the tallest midget.” It didn’t matter which was a little better, they all were the wrong answer. The technical term was that we had found a “local maximum.”
Further exploration bore this out: when we tested what happened if we enabled watching on TVs through XBox, Playstation, Wii and Roku — that was our Denali moment — it literally blew away the puny earlier results. The button stuff was all the foothills.
At Netflix in those days, we didn’t have time for incremental improvement — we were instructed to swing for the fences, only significant improvements, which might only be found in radical, off-beat, even insane idea exploration. Big wins and, often, big misses. As we learned, modest success was met with a generous severance package. When I look at product testing and iteration in new products at other companies, I see lots of messing around in foothills, that at worst is just ‘rearranging deck chairs on the Titanic.’ Test big ideas. You’re looking for Denali.
You know it when you get the right one. It’s clear. Particularly early on, particularly with smaller sample sizes, the thing you’re looking for is not 10% better, but 10X better than the others.
Incidentally, this happened to me last week. I’ve been writing stories on Medium for a couple months now. Most lumber along with some readers, dozens or maybe even hundreds, and occasionally there’s a big one, a spike, hitting many hundreds in a day. So that was my world — the ones that didn’t “perform” and the few that did. But last week a story did something different, it had thousands of reads, and increased over subsequent days. And all the data on the others turned into the noise, and there was Denali.
I know it’s all relative, but if you’re a data hound remember that you’ll usually know it when something works.
