10x v 10% impact: Successful product owners focus on what to build, less on how it’s executed

Businesses use interventions to optimize performance.

Companies and product owners start projects when they want to change something about the way in which they are doing business. Typically, a team is tasked to build and ship a product feature to solve a particular customer problem. And as the product owner, you typically try to maximize the impact of your team’s solution.

So, how do you make sure that your team is building the right thing? And, how do you know that the team is building the right thing perfectly? Also, how do you validate that your team has the intended impact?

I’m sure you can easily list a couple of projects for which you’ve great answers to those questions. But what about all the remaining projects? What do they have in common and why did you not deliver the indeed impact?

Selecting an intervention’s concept, optimizing it perfectly and measuring its impact.

My framework enables PMs to have reasoned answers to these questions while maximizing the likelihood of building an impactful product feature. At its core, conceptual optimization distinguishes the framework of identifying the best solution from executing it perfectly and measuring its impact:

1. Concept selection: Use your understanding of the customer need to identify all conceptual solutions to a problem and identify the best conceptual solution through insights
2. Concept optimization: Work with the subject matter experts to create a minimal viable prototype product feature and iterate on its execution based on customer feedback
3. Measure effectiveness: Run a field experiment to test your product feature’s impact on the target variable prior to rolling it out to all users

To talk more about the framework, imagine the following scenario: You are the PM for a health app and you’re tasked with increasing your users average weekly running frequency from once a week to three times a week.

So, how will this specifically play out in the conceptual optimization framework?

Concept selection

Your goal is to identify the key behavior change components that will enable your customers to change their running patterns. Based on customer interviews (10-ish customers with different running frequencies to reach insight saturation in each group), customer surveys (to validate the insights from the interviews), past research insights and secondary research insights, you can build a detailed understanding of what it takes to change the running behavior. Based on those insights, you shortlist your long list of what you could build to the key requirements for a seven day challenge. For example, you ask your team to include motivational components — such as daily streaks, and leaderboards — and ability-focussed aspects such as animation-based ‘how to run’-tutorials, and lists of personalized local running routes. Note, that based on the research insights you decided not to include other possible components such as monetary rewards or video-based ‘how to run’-tutorials.

Concept optimization

Secondly, you and your product team build a solution based on your requirements from concept selection. Copy, design, data science and engineering team members build the initial prototype of the feature. Through iterative prototyping, you and your team optimize the prototype and take it to its final high fidelity execution. Note, that the customers that are providing usability feedback are not the same customers that helped you identify what to do in the first place. At this stage you’re no longer looking for feedback on what to do but solely on how to do it best.

Measure effectiveness

Lastly, you and your team measure the impact of the conceptual solution you’ve built. Using the gold standard of A/B testing, you conduct a causal experiment: the control group receives the usual experience while your treatment group receives the running challenge product feature the team built. The goal is to test if your product feature increases your users running frequency by two weekly runs (Control: one weekly run, Treatment: 3 weekly runs).

Interpretation of impact results

If the treatment group runs significantly more than the control group, the team has reached great success. In the alternative case, the question becomes why did the product feature not increase the running frequency? To answer this question, conceptual optimization points at concept selection. It’s most likely that something went wrong when collecting the insights: maybe customers didn’t accurately recall how they were able to change their behavior, or the team didn’t validate the interview insights with a survey. And concept optimization is unlikely the reason for failure: subject matter experts built the solution with customers’ usability feedback and it’s unlikely that the feature would have performed significantly better with tweaks to the copy or the visuals. In this example, the challenge might have been a success if the concept had included a monetary reward.

Distinguishing concept selection from concept execution is paramount.

In my experience, the key is to ensure that a product team has identified the right things to do. When I draw my framework, I highlight this by making all reasonable executions of conceptual solution 3 (within the orange circle) significantly better than all reasonable executions of conceptual solutions 1 and 2. The main outcome and goal of this framework is to identify conceptual solution 3 and its components.

Selecting the best concept from all concepts, executing it perfectly and measuring its impact.

This illustration also highlights that a project’s impact ceiling is decided very early on. To achieve a 10x impact (conceptual solution 3), you need to identify it and its components early on. If you fail to identify conceptual solution 3, your impact must be substantially lower: No matter how well your team executes conceptual solution 1 or 2, it’ll always be less impactful than all reasonable executions of conceptual solution 3. If conceptual solution 2 is selected for execution, the impact will be 10% when it could have been 10x.

Focussing on what to build first and on how it’s executed afterwards.

To summarize, these are the six steps I use to make sure I’m building the right thing perfectly and they will also help you identify the ideal execution of the best conceptual solution:

1. Identify all possible conceptual solutions that may address the underlying problem
2. Narrow these conceptual solutions down to the one you want to build
3. Execute the selected conceptual solution to your best ability
4. Conduct usability research to optimize its execution to the max
5. Measure the impact of your successful product feature
6. Celebrate your and your team’s success :)

Hey! If you found this post actionable or insightful, you can find more posts on my personal blog.