Discovery Metrics: How to quantify the effectiveness of your discovery efforts

Photo by Noble Mitchell on Unsplash

The best Product teams test several hypotheses a week in their relentless pursuit of reducing the key value, usability, feasibility and business risks in their product. But how can you instil this mindset into your team and how do you know whether this is driving value for your product?

In December I was featured by Teresa Torres in a ProductTalk article about how we at SuperAwesome have implemented Teresa’s opportunity-solution tree. In this follow-up article, I explain how we have taken this further and found a way to quantify the impact it is having on the effectiveness of our product discovery.

Making use of the Opportunity-Solution Tree

I won’t go into detail about the opportunity-solution tree here as this is covered in the previous article. In short, it is a great way to organise your thinking around the opportunities identified by your team to deliver on a specific measurable outcome for your product.

Beyond this, it enables you to multi-track different solutions that deliver on each opportunity whilst keeping sight of the original outcome, and the relationships between outcomes, opportunities and solutions. The output of the tree is a set of experiments that the team can run in order to test their assumptions around each solution. The results of these experiments can be used to compare and contrast the different solutions, and ultimately find the best way to deliver on the chosen opportunity and drive the desired outcome.

Our adaptation of the Opportunity-Solution Tree at SuperAwesome

While using this great tool at SuperAwesome, we are continuing to iterate on the model and have adapted it in various ways across our product teams. This has involved adapting it into a spreadsheet format, for those who prefer a tabular layout, that can easily be filtered and restructured as opposed to the more visual tree diagram.

We have also added an “assumptions” section to the tree between “solutions” and “experiments”. As I explained in the ProductTalk article, this is because we want to be more thorough when breaking down our solutions into the underlying assumptions that need to be validated in order to be confident that we are addressing the opportunity in the best way possible. We sometimes felt that we needed to build the feature and AB test it in order to learn whether it was successful in driving improvement in a particular product KPI and wanted to ensure we were minimising the effort we were taking to invalidate our assumptions. The obvious problem here was that this effort would be wasted if we found that the solution did not achieve the desired outcome.

Regardless of the chosen structure of the tree or how different teams choose to visualise it, the important thing to remember is that the underlying process remains the same. I’ve started to think of this process as opportunity analysis. This is because we tend to be good at coming up with solutions that we believe will drive outcomes but taking time to better understand the problem space can often reveal alternative solutions that are less complex or better meet the needs of our users. In this way we might find a more effective solution that we would have overlooked, had we not carried out an opportunity analysis.

Analysing the opportunity in this way is what makes this process really valuable. It teaches us to scrutinise the opportunity as much as we would the solution which helps drive our thinking around the assumptions that need to be tested.

So what do we measure?

Taking this a step further, we wanted to find a way to measure the throughput and the quality of our discovery processes. This would give us an indication of whether our use of the opportunity-solution tree was having the desired effect. We reasoned that we do this for our delivery phase of development, and so it made sense to apply the same approach to discovery. For delivery, we use metrics identified by Gene Kim, Jez Humble, and Nicole Forsgren, the authors of the book “Accelerate: The Science of Lean Software and Devops” and the associated State of DevOps reports. Their research has shown these metrics to be predictive of organisational performance (profitability, market share, productivity, etc).

These delivery metrics are “mean time to restore” (MTTR) and “failure rate” as a measure of quality as well as “lead time” and “deployment frequency” as a measure of throughput. Using these metrics has helped to dispel the myth that a product team has to find a compromise between quality and throughput. Instead, they promote the goal of improvement in both of these areas through following certain best practices.

We applied a similar model to our discovery in order to ensure that we are making the best decisions about what features would deliver the most value to our customers (quality) as well as maximising the speed at which we learn about our users’ needs and their behaviour (throughput).

We introduced two new metrics as part of our overall team metrics. These were “number of tested assumptions per sprint” (throughput) and “shipped measured value per week” (quality). The “tested assumptions” are a measure of how many learnings we have taken from our experimentation every sprint; “shipped measured value” measures how many quantifiably valuable product improvements were shipped. The difference in time frame here was to support continuous delivery and ensure that we were releasing throughout the sprint.

We measure these through catch ups once a sprint between myself and the Product Manager from each team. We talk through the “shipped measured value” items and try to answer the following four questions for each item:

1. What is the feature / change / improvement?
2. What is the value that we expect to gain from it?
3. How have we measured success?
4. What was the result?

Once we have confirmed that the result shows that the change has delivered on the desired outcome, then it can be counted as an item of “measured value”. We do not take into account the size, effort or impact of a feature as part of this metric although these are considered in the questions above. Ultimately the goal is to ensure that the product improvements we make result in measurable value but we don’t want to get bogged down with assigning subjective weightings to different items.

We follow a similar approach with “tested assumptions”, this time answering the following four questions:

1. What was the assumption / hypothesis to be tested?
2. How did we test it?
3. What was the result?
4. Did the result validate or invalidate our assumption?

Once we have adequately answered these questions we can count this as one “tested assumption”. The important difference to note here is that when we look at “tested assumptions” as a measure of throughput, the assumption does not need to be validated to be counted since, regardless of the result, we have still learnt something.

These measures are then all captured in a spreadsheet and visualised in a chart along with the targets we have set for each metric. This gives each product team visibility of how they are doing against the target. We also share our successes and failures in a cross-team peer review session, where we can challenge each other to better our process over time. The objective is to encourage the teams to find quicker, cheaper ways of running experiments. Ideally, this would be done without writing any code at all where possible. Then at the same time we want to ensure that our teams are testing the most risky assumptions in order to develop truly valuable features.

What impact is this having?

Our use of the opportunity-solution tree has contributed towards a mindset shift across our product teams towards running multiple experiments in parallel that require little or no development work. This has also strengthened our collaborative approach to product discovery, with Engineers, Designers and Product Managers all contributing to the product decisions being made as part of our generative culture of rapid experimentation and collaborative learning.

Our use of discovery metrics in combination with the above, has allowed us to quantify the impact the changes are having. What we have seen is a threefold increase in the number of experiments each team is running per sprint as well as a steady increase in the amount of measurable value that we are able to deliver to our customers.

Want to come aid our discovery efforts, and make the internet safer for kids? We are hiring!