Responsibility #3: Define Success and Measure Performance Over Time

By Michael Colemere and Candianne Haacke

The Benefits of Product Success Criteria and Performance Measures
Product managers are responsible for defining success criteria and measuring performance over time. Success criteria are the documented standards by which stakeholders will judge a product. Defining success at the beginning of product development is important because it forms the basis for future decision making. Measuring performance is important because it indicates the extent to which success is being realized.

Clearly defined success criteria are a tool product managers can use to know when to say “yes” or “no” to new product requirements. They are also useful in tracking progress toward the end goals.

Example of User-Centered Success Criteria
So, how do product managers define success? They begin by identifying the audience’s pain, problem, or “jobs to be done” (as described in this short video by Clayton Christensen). Then, they measure the extent to which their product addresses that pain, problem, or “jobs to be done.”

For example, The Church of Jesus Christ of Latter-day Saints recently broadcast their twice-a-year general conference on their flagship website, Instead of linking to a stream deep within the broadcast section of their site (like they had in several previous years), they embedded a live feed directly on the homepage — an experience similar to Apple’s press conferences. This change occurred as a result of clearly defined success criteria: specifically, providing quick and easy access for viewers. If the success criteria were based on traffic volume to the previous broadcast section, the change might not have been made and the user experience would not have improved.

Author Gerry McGovern helps product managers define success criteria (for websites) by asking the following three questions.

1. Are you an organization that is genuinely customer centric and likes to serve? If you are, then your website will be focused on allowing your customers to complete their tasks as quickly and easily as possible. Even if your organization is complex, your website will be simple, because you will work hard to hide your internal complexity.

2. Are you an organization that thrives on control and hierarchy? Then your website will be all tell and sell. It will be all about telling customers what you want them to hear, about selling them what you want to sell them. It will be about messaging, communications, PR, and marketing.

3. Is your organization overflowing with vanity and ego? Then your website will be full of pictures of managers shaking hands. It will be fully of glossy images of attractive actors pretending to be happy customers. It will be full of meaningless jargon, and you will constantly be telling your customers how much you care about them.

In the general conference example, key performance metrics were still captured, just not at the viewer’s expense. In fact, significant strides have been made by the product manager, Rob Jex, to establish common metrics to monitor live event performance. To achieve common performance indicators, Rob has had to break his big idea into chunks, seek consensus, figure out the logistical how-to’s, and accept some short-term redundancy.

Now that he is two years into achieving common measures, a cross-functional team meets monthly to compare event performance. They discuss lessons learned and explore improvements that could be made (improvements to lighting, relevant b-roll, effective ways of distributing streamed media through third parties, and more). The benefits to establishing common performance measures, in this case, have led to greater collaboration and innovation.

Example of Performance Measurement
Pearson’s law states, “When performance is measured, performance is improved. When performance is measured and reported back, the rate of improvement accelerates.”

So, how does a product manager measure performance? One practical tip is to read Douglas Hubbard’s book titled “How to Measure Anything.” Another tremendous resource published by Google is “Measure What Matters Most.” When Rob Jex began coordinating live event publishing performance metrics, his biggest challenge was just trying to keep it simple. He found they could literally measure anything, but they didn’t need to. Once they defined the core metrics needed, they had enough information to start looking at the “why’s“ and “what’s” of what was happening during events.

In summary, success is defined by the way it helps people and not on features, hierarchy, or ego. It is about identifying the audience’s pain, problem, or “jobs to be done,” and then measuring results over time. In doing so, your product offering will be much more relevant in people’s daily lives. When questions arise about why your product exists, the answer will be clear in terms of purpose, impact, and performance over time.