Product Managers are Treasure Hunters
And Experimentation is our Metal Detector
You’ve probably heard this line before: “The PM is the CEO of the product.” There’s a lot of truth to it. Like CEOs, we’re ultimately accountable for the success of the products we manage, and we succeed by unlocking the potential of the team we serve. But I think that more often than not, this metaphor is misunderstood. We hear “CEO” and we picture ourselves delivering orders to lowly subordinates, ringing the NASDAQ bell as confetti falls, and then riding our helicopters off into the sunset.
Contrast this with the messy reality of product management: a daily grind of asking questions, responding to emails, running between people’s desks, making sense of graphs, triaging bugs, responding to more emails, making slides, getting interrupted, and then responding to even more emails. And did I mention the endless cadence of stand-ups, 1:1s, and planning meetings that fill every PM’s calendar to the brim? The day-to-day reality of product management is much less glamorous than the CEO metaphor implies. (Though I doubt a real CEO’s day is all that much better.)
The CEO metaphor isn’t wrong, but it’s so easily misunderstood. It highlights the power and prestige of the title while obscuring the actual substance of the job, and more importantly the purpose. Why do both PMs and CEOs spend most of their time meeting, listening, and writing? What does it all add up to? As Intel CEO Andy Grove puts it in one of my favorite PM books, High Output Management:
What we actually do is hard to pin down and sum up. Much of it often seems so inconsequential that our position in the business hardly seems justified. Part of the problem here stems from the distinction between our activities, which is what we actually do, and our output, which is what we achieve. The latter seems important, significant, and worthwhile. The former often seems trivial, insignificant, and messy. But a surgeon whose output is a cured patient spends his time scrubbing and cutting and suturing, and this hardly sounds very respectable either.
So what does all our scrubbing and cutting add up to? Who’s the patient that we’re curing, and what’s the disease?
I would humbly answer: the stunning waste involved in almost all product development. At its root, our job is about steering a path through a wasteland of opportunity cost.
The product development wasteland.
Most software engineering is a waste of time. This sounds dramatic, but it’s the essence of why product management exists as a discipline.
Most products barely get used. Look at the App Store. There are over a million apps available to download, most of them built at significant engineering and design cost. But the vast majority of these never take off. Products follow extreme power laws: the top 0.1% get all the usage and earn all the money, while the other 99.9% sit forgotten. Some of those 999 million apps are barely designed, or badly engineered, or poorly marketed. But what the top 1000 have in common is that they each solve an essential problem, and they do it well. Their product matters to people in a way that most others don’t. This is our purpose: to build the tiny minority of products that matter most. (To be clear, these aren’t just the biggest products. There’s value in the long tail too — but even there, in any niche, the same power law applies.)
Now take a deeper look at the most successful products, the 0.1%. Even there, you’ll find that not every feature in those products has the same impact. And in fact, it’s the same distribution: most features barely get used, and just a handful of them are key to success.
For example, Gmail is one of the most successful technology products of all time, with more than a billion active users. That’s more than the next 999 email apps combined. What drove that success? I’d argue it was just a handful of differentiating features: high capacity storage, good search, and maybe conversation threading. Gmail’s success came from nailing those features and leaving out the bloat of everything else. Fun fact: when Gmail launched, it didn’t even have rich text formatting, let alone folders or read receipts or any of the other 997 features you might find in a competitor’s checklist.
And all of this still doesn’t capture the extent of the waste, because we’re only talking about products and features that have actually launched. What about the code that gets rewritten five times because the objectives keep changing? What about the project that’s axed six months into development because everyone forgot why they started it? What about the thousands of startups that sink into oblivion before ever launching a product because they never bothered to show it to customers? An unbelievable amount of collective engineering time is lost to the black hole of product confusion every year.
Finally, add to all this the idea of opportunity cost: even when you build a great feature on a successful product, you always have to ask yourself, could the same time have been spent on something even bigger? Is this really the highest calling of my team?
Once you grasp the sheer scale of this problem — canceled launches plus forgotten features plus failed products times opportunity cost — you start to realize that building great products is like searching for buried treasure hidden in a wasteland of mediocre ideas. Somewhere in that vast expanse, there are incredible gems to be found. The products that users treasure, the ones that massive, world-changing business are built on. And the trillion dollar question is: how do we make more of those, and less of everything else?
The PM’s job is to find the treasure.
Our objective as PMs is to find these products and features. We need to find the rare problems that truly matter for our users and the business, mark them clearly on a map, and then rally our teams to get there and dig until we find the right solution.
Finding these gems is not a glamorous exercise. Like hunting for gold or oil, it’s a mundane process much of the time: prospecting for users to interview, drilling for meaningful insights, poring over charts looking for the sparkle of something interesting. Most of all it means lots of useless digging. You have to build hundreds of mock-ups, prototypes, and experiments, and then watch most of them fail, just to find a single solid gold feature.
The best product managers are the ones with the best treasure maps. They have deep empathy for their users, mastery of their analytics, and they experiment constantly to hone their understanding of the landscape. By combining these insights, they develop strong hypotheses about where the treasure is hidden — and then they can mark it with an X on a map. That’s all a product roadmap is, really. It’s a sequence of places to dig, prioritized by potential (what kind of treasure is waiting there) and effort (how deep will we have to shovel).
Now if you’re caught in the grip of the CEO metaphor, you might think that’s all there is. You declare “X marks the spot!”, and your job is done. Your team will shovel away and you’ll sit back with a piña colada supervising. And if that’s what you believe, then you’ll make a terrible treasure hunter.
The truth is that your job as a PM has only just begun. Now you have to convince everyone else that you’ve found something, that your map is accurate, and that the treasure at the bottom of the hole will really be worth it. The core of a product manager’s work is persuasion. You may see the treasure map clearly, but if you don’t help your team see the same thing then it hardly matters. Your role is to lead your team on a journey, with incremental launches that deliver tangible value every step of the way. You have to be digging for treasure constantly, building new prototypes and features in search of that top 0.1% idea that can bend the trajectory of your product.
Once in a while, you’ll find that gem by chance, stumbling into product-market fit by accident. Many startups begin this way, with a simple idea that takes off faster than the founders ever imagined. But I’m more interested in how to reliably find that treasure. How can we invent the second product, and the third? How can we turn product management into a repeatable process? To do that, we can’t be blind treasure hunters, digging holes at random and crossing ourfingers. We need a tool that can steer us in the right direction, tell us how far to dig, and evaluate what we’ve found at the bottom.
Experimentation is our metal detector.
Product managers have many tools for steering product development. We can interview users, run A/B tests, analyze historical data, perform gradual rollouts, and run clever studies like painted door and wizard of oz. My intent here isn’t to advocate for any one approach in particular, because each has its place. I’m more interested in what they all have in common, which is the broad theme of experimentation. By “experimentation,” I mean exposing an idea to your users to understand how they’ll use it and what value it provides. Experimentation means eliminating guesswork by testing things out in the wild as early as possible.
If PMs are treasure hunters, then experiments are our metal detectors. Taken together, these techniques are the most powerful tools we have for reliably avoiding waste and discovering opportunities. That’s because they can systematically guide us through each stage of the product development treasure hunt.
- Experiments tell us where to dig. The best place to start a new project is always generative research, building up deep empathy for your customer and the world they inhabit. This gives you the map. The next stage is understanding that user’s problems, digging not just for interest but proven intent to solve. This gives you a general direction. The final step is to explore solutions, defining through a process of design and engineering how your product can uniquely solve those problems. This last piece is the “X” that marks the spot, and the best way to pin it down is through constant experiments to validate ideas as simply and quickly as possible.
- Experiments help us steer. Once we have an idea, we could just go build it, but then we risk screwing up the details that make or break our feature. We know our customers are interested in sports content. But do they want it in video form, or written articles? Do they prefer editorial or news? How do ads impact the experience? Nailing a product launch is about tweaking a hundred different variables to zero in on the perfect balance. Getting any of these wrong can defeat the whole project, like digging a hole just inches away from a treasure chest and finding nothing but dirt. A/B/n and multivariate testing help us build many different variations and find the ideal combination.
- Experiments tell us how deep to dig. Product management isn’t just about prioritizing features to work on. It’s just as much about deciding scope and recognizing when the perfect is the enemy of the good. Which functionality needs to be easy, and which needs to be merely possible? In my experience, you can always cut scope further, but only if you have a deep understanding of your users. Techniques like staged rollouts and iterative A/B testing let you quantify the value of a minimal feature set without having to build the whole thing. Even more valuably, these experiments can tell you when to quit early, buying time to chase other opportunities rather than getting caught in an ever-expanding hole.
- Experiments find landmines. One of the biggest risks in product development is chasing an idea that seems appealing but actually makes your product much worse. For example, we might build a social feature in the hopes of driving engagement, only to find that it completely alienates our users. Or we might redesign our navigation, but then hopelessly confuse people in the process. We might even nail the design, but make a simple engineering mistake that degrades performance or causes a crash. Experimentation techniques like feature flagging and gradual rollouts protect against this downside risk, ensuring that we stop digging before we hit trouble and have the chance to correct.
- Experiments quantify impact. Even when we nail all the pieces above — choosing the right problem, scoping it effectively, validating success, and iterating to perfection — we still can’t be sure we’ve found the right treasure. Sure, we hit something yellowish and shiny, but is it gold? The final gift of experimentation is its power to tie the work we do to business impact. Through A/B testing and careful analysis, we can measure how each product launch directly contributes to our company’s ultimate goals, whether that’s revenue or readership or activity. And this means we not only make an impact, but also gain the data we need to justify bigger and more ambitious projects. If you want a bigger pirate crew and a nicer boat, you have to prove to the business that you can find real gold.
This process isn’t just a linear path, it’s a cycle. Product managers succeed or fail through persuasion, and our biggest asset is our credibility. If our teams trust us, from the newest developer to the top stakeholder, then we have the freedom to take big risks and build world-changing products. The moment that trust erodes, we’re stuck making incremental tweaks and arguing every detail of every change. This credibility has to be earned and re-earned constantly. We gain it by proving that we have our fingers on the pulse of our users, and we lose it by sending our teams on wild goose chases. We gain it by rolling up our sleeves to dig along with them, and we lose it by leaving them to shovel alone. And most importantly, we gain credibility by finding treasure. Every successful launch empowers us to think bigger and push farther.
So if you’re an aspiring product manager, or an experienced PM who wants to be even better, don’t think of yourself as a CEO. Instead, embrace your role as a treasure hunter. Your job is to lead your team on a journey through a wasteland of useless features and bloated apps, in search of impactful work that can transform your entire business. And the secret to thriving on this journey is experimentation. An experimental mindset will pull you towards success, guiding you like a metal detector straight to the biggest opportunities. It’s a journey where you don’t need to have all the answers, you just need to run the right experiments.
This post is part of a continuing series on building better products through constant experimentation. Our goal is to demystify the process of A/B testing and feature rollouts, so please share your questions and feedback in the comments. And if you’re ready to take your product experimentation to the next level, check out Optimizely Full Stack.
