Don’t Trust Me, I’m Just the Product Owner
Lessons learned from my biggest product mistake
More than 10,000 sessions in our customer portal, over about 6 weeks — and only four people had set up notifications.
Crap, I thought. This is bad.
Rewind six months.
The dev team and I had just wrapped up a massive, nearly-flawless data migration. We’d shifted almost 20,000 servers, firewalls, load balancers, and other bits and pieces from a nearly decade-old monitoring system, into a brand new platform.
Unbelievably, only a handful of minor issues had cropped up, all of which were easily dealt with. Our beta users were happy, and management was ecstatic. We were feeling pretty good.
At the same time, we had also built a set of tools in our customer portal to let users browse their monitoring data. The idea was that power users could access their data directly in the monitoring platform, but our less savvy customers could use the simplified tools in the portal.
The feature was a masterclass in technical simplicity. Metrics from over two dozen different devices and applications were distilled down into a simple graph. We handled different sampling rates, different units, even missing data with ease. All the corner cases were covered.
And — most importantly — it looked sexy as hell.
So eventually the time came to pick the team’s next project. One of the main things I knew our customers hated about the old system the most was the lack of notifications. Quite simply, there were none. Most of the time, they found out their systems were down when their own customers started complaining. Not good.
Fresh off our success with the simplified approach, I decided we should continue down the same path. Power users could use the built-in notifications in the monitoring platform, and we’d build a no-frills version for everyone else.
There was just one giant hurdle to overcome. The monitoring platform didn’t expose their notification system via the API — we’d have to roll our own. This meant the feature would take much longer than anticipated.
Cue the montage.
Three or four months later, we were ready to ship.
Given the lack of an API, we had done a pretty good job. The user could pick a metric, and enter their email address. We’d automatically send them an email whenever the metric went outside the defined thresholds, and then another email when it returned to a normal range.
We’d received scattered feedback from the beta users that everything worked as expected, and wrote an FAQ to answer the few questions that came up. Support was trained, release notes were written, analytics were wired in, and we were ready to rock and roll. We hit the go button, smoke-tested, and everything seemed fine.
Since we were already way outside our original schedule, I shifted the team onto another project, and nearly forgot about the notifications feature.
A few weeks later, though, I decided to log into our analytics, to see how the feature was fairing.
Out of over 10,000 sessions in our customer portal, only four people had set up notifications. Not good — not good at all.
After talking with some of the customers that had used the feature, the problem became obvious.
They didn’t want a simple notification system.
In fact, they didn’t want notifications at all.
They just wanted their servers to keep working. And they wanted us to fix them when they stopped working.
When a server goes down at 3 o’clock in the morning, the last thing they wanted was an email saying “Hey guys, your shit’s broken — maybe you should go fix it”. What they wanted was an email at 3:15am saying “Hey guys, your server went down, but we fixed it for you without having to wake you up”.
All we’d built was a prettier way to let them know things were broken. We weren’t solving the problem at all, and so they weren’t using the feature. And on top of it all, we wasted three months of development time.
The unfortunate fact is, what customers say they want is almost never what they actually need. Our beta users told us they wanted automatic notifications when their systems went down — but instead, they had hired us to take care of this for them, so they didn’t have to wake up and do it themselves.
We simply weren’t helping them complete the job that they hired us to do.
Clayton Christensen wrote about this in detail in the Harvard Business Review. He gives a specific example of a fast food restaurant that wanted to increase sales of one particular product — a milkshake. The restaurant had tried many things — different flavours, prices, and thicknesses — but none of their ideas increased sales.
Eventually they hired a new researcher, who spent a day in the restaurant to try and understand the job that customers were hiring the milkshake for:
He was surprised to find that 40 percent of all milk shakes were purchased in the early morning. Most often, these early-morning customers were alone; they did not buy anything else; and they consumed their shakes in their cars.
The researcher then returned to interview the morning customers as they left the restaurant, shake in hand, in an effort to understand what caused them to hire a milk shake. Most bought it to do a similar job: They faced a long, boring commute and needed something to make the drive more interesting. They weren’t yet hungry but knew that they would be by 10 a.m.; they wanted to consume something now that would stave off hunger until noon. And they faced constraints: They were in a hurry, they were wearing work clothes, and they had (at most) one free hand.
After analyzing the data, the restaurant made their shakes thicker, and also added chunks of fruit for a little unpredictability. Sure enough, sales skyrocketed.
The customer wasn’t buying a milkshake because they liked the flavour, or because they were cheaper than their competitors. They were hiring the milkshake to make their commute more interesting, and make them less hungry throughout the morning.
The critical point that the researcher realized was to ask the customers questions about what they do and why, and not about the milkshake itself.
Customers don’t want to buy your product. They want to hire your product or service to get a job done.
In our case, the job they needed done was to bring their servers back online — not to let them know they needed to do it themselves. As easy-to-use as it was, our notifications feature simply wasn’t helping them do their job — so they weren’t using it.
Of course, hindsight is 20:20 — we could have figured this out by observing how our customers were using the existing system, or simply by asking them. But, my assumption — that what they told me they needed would actually solve the problem — proved terribly wrong.
So next time you’re working on a new product, or feature, be sure to understand the job your customers are hiring your product for.
Oh, and if you’re a developer? Be sure to kick your product owner’s ass if they suggest spending three months on a feature without a damn good reason.
Like what you just read? Be sure to click on the green heart below to so other people can enjoy it as well!