Concepts of Prioritization — Chapter 6

Chapter 6 — Balancing Cost of Research vs Value of Information

6.1 Overview of the Tradeoff Between Research Cost and Expected Benefit

6.2 Methodologies for Estimating Research Costs

6.3 Decision making in uncertainty

6.4 Estimating Value of information

There are three classes of decision making situations.

• One is a class of deep uncertainty, where uncertain scenarios occur randomly and in which we have absolutely no understanding of the factors contributing to the randomness.
• The second class is the class of risky problems in which we assume to have complete understanding of the factors contributing to the uncertainty, and hence we can develop a theoretical probability distribution of the occurence of different scenarios.
• There is a third class , which is in between the first two, where the decision maker knows enough about the scenarios to know that they are not absolutely random, but not enough to analyze them fully and come up with a theoretical distribution to predict their occurence.

In every situation of decision making, the problem lies in uncertainty, if there is no uncertainty, then there is no decision to make, it is a logical choice. Information helps one to improve the success of the decision made if the information is utilised to reduce uncertainty in the outcomes of the decision made. In chapter 2, we introduced cost of measurement and value of information. In this chapter, we deep dive into balancing these two. To put it simply, the value of information derived from any specific research should be greater than the cost incurred for the research.

6.1 Overview of the Tradeoff Between Research Cost and Expected Benefit

Research plays a critical role in product development by supplying valuable insights and data that enables decision-making, shape product features, and contribute to the overall success of the product. However, research activities are not without costs, which can include time, effort, resources, and financial expenses. Consequently, product managers face the challenge of balancing the cost of research against its anticipated benefits, aiming to optimize the value derived from research investments.

The tradeoff between research cost and expected benefit can be conceptualized as an optimization problem in which the goal is to maximize the return on investment (ROI) from research activities. To achieve this, product managers must consider various factors, such as the scope of the research, the resources allocated to it, and the likelihood of achieving the desired outcomes.

To illustrate this tradeoff, let’s consider an example.

Imagine a product manager at a software company who must decide whether to invest in usability research for their product. The research cost includes hiring a dedicated team of researchers, conducting user interviews, and purchasing equipment for testing. On the other hand, the expected benefits involve improved user satisfaction, reduced support costs, and increased customer retention.

To make an informed decision, the product manager must weigh the costs against the potential benefits. They could calculate the ROI by dividing the estimated benefits by the total costs of the research. For instance, if the research cost is $50,000, and the anticipated benefits are worth $200,000, the ROI would be:

ROI = (Expected Benefits / Research Cost) * 100

ROI = ($200,000 / $50,000) * 100 = 400%

In this case, the product manager might decide that the potential 400% return on investment justifies the research cost. However, it is crucial to acknowledge that these calculations are based on estimations, and the actual outcomes may differ. Moreover, in real-world situations, the tradeoff analysis can be more complex, involving multiple research projects, competing priorities, and various constraints.

By understanding the tradeoff between research cost and expected benefit, product managers can make strategic decisions that maximize the value derived from research activities, ultimately contributing to the success of their products.

6.2 Methodologies for Estimating Research Costs

Accurately estimating research costs is a vital component of the cost-benefit analysis process in product management. Several methods can be employed to estimate research costs, each with its advantages and limitations. Below, we discuss three common approaches and provide examples for each.

1. Time-based estimation

This method involves calculating the time required to complete research activities and multiplying it by the hourly rate of the researchers involved. Time-based estimation helps capture the human resource costs associated with research projects.

Example: Suppose a product manager plans to conduct a user research study that will take 100 hours to complete. If the hourly rate of the researchers involved is $50, the total cost of the research project would be:

Research Cost = Total Hours * Hourly Rate Research Cost = 100 hours * $50 = $5,000

The time also adds on to the time to market, the cost of which is calculated as cost of delay, which we will explore more in detail in future chapters.

2. Resource-based estimation

This approach identifies the resources required for a research project (e.g., equipment, software, facilities) and calculates their associated costs. Resource-based estimation enables product managers to account for the tangible assets needed to execute research activities.

Example: A product manager needs to conduct a series of focus groups for a new product. The required resources include renting a meeting room for $500, providing refreshments for $200, purchasing recording equipment for $1,000, and acquiring a transcription software subscription for $100. The total cost of the research project would be:

Research Cost = Meeting Room + Refreshments + Equipment + Software Research Cost = $500 + $200 + $1,000 + $100 = $1,800

3. Expert judgment

This method relies on the expertise of subject matter experts to estimate the cost of research activities. Expert judgment is particularly useful when a project involves novel or complex research methodologies, and the costs are difficult to quantify using other methods.

Example: A product manager is considering a cutting-edge research technique to study user behavior but is unsure about the associated costs. They consult with a renowned expert in the field, who estimates that the research project will cost approximately $10,000, based on their experience with similar projects and knowledge of the required resources.

By employing these methods to estimate research costs, product managers can create a more accurate representation of the investment needed for research activities. This information is essential for informed decision-making and effective cost-benefit analyses in product management.

Estimating the expected benefits of research activities is crucial for determining the potential return on investment and making informed decisions about research priorities. Several techniques can help product managers estimate expected benefits, taking into account various factors such as future cash flows, probabilities of outcomes, and the time value of money. Fundamentally, research can help in understanding the probability of a certain value of outcome

In the next section we discuss how we can make decisions based on value of outcome.

6.3 Decision making in uncertainty

Class one of uncertainty deals with situations where uncertain scenarios occur randomly, and we have no understanding of the factors contributing to the randomness. In such cases, it is challenging to derive probabilities for different outcomes. Therefore, traditional probabilistic approaches might not be suitable for decision-making in this class of uncertainty.

In such situations, alternative decision-making approaches can be employed, such as the maximax, minimax, or minimax regret criteria. These approaches do not rely on probabilities but rather focus on potential outcomes to guide decision-making.

1. Maximax (Optimistic) Approach: The maximax approach is an optimistic decision-making criterion. It assumes that the best possible outcome will occur for each decision alternative. The decision-maker selects the alternative with the highest potential payoff, essentially maximizing the maximum payoff.
2. Minimax (Pessimistic) Approach: The minimax approach is a pessimistic decision-making criterion. It assumes that the worst possible outcome will occur for each decision alternative. The decision-maker selects the alternative that minimizes the worst possible outcome, essentially minimizing the maximum loss.
3. Minimax Regret Approach: The minimax regret approach focuses on minimizing the regret associated with each decision alternative. Regret is the difference between the best possible outcome and the actual outcome under each scenario. The decision-maker selects the alternative that minimizes the maximum regret, essentially considering the opportunity cost of not choosing the best alternative in each scenario.

In class one of uncertainty, where probabilities of outcomes are not available, these approaches can help guide decision-making by focusing on the potential outcomes rather than the likelihood of their occurrence. However, it is essential to note that these approaches come with their own set of limitations and biases, as they are based on either extreme optimism, extreme pessimism, or regret minimization. Decision-makers should carefully consider these biases when employing these approaches in real-world situations.

Example:

Suppose you are a product manager, and you are facing a prioritization challenge for an upcoming sprint. There are three potential features (A, B, and C) to implement, but you cannot determine the probabilities of their success. Each feature has different expected outcomes in terms of user engagement increase:

• Feature A: Best case — 100 users, Worst case — 20 users
• Feature B: Best case — 80 users, Worst case — 50 users
• Feature C: Best case — 60 users, Worst case — 40 users

Using the three decision-making approaches, you can prioritize the features based on the potential outcomes:

1. Maximax (Optimistic) Approach: Here, you focus on the best possible outcomes. The maximum engagement increase for each feature is:

• Feature A: 100 users
• Feature B: 80 users
• Feature C: 60 users

With the maximax approach, you would prioritize Feature A, as it has the highest potential engagement increase.

2. Minimax (Pessimistic) Approach: In this approach, you focus on the worst possible outcomes. The minimum engagement increase for each feature is:

• Feature A: 20 users
• Feature B: 50 users
• Feature C: 40 users

With the minimax approach, you would prioritize Feature B, as it has the least worst outcome in terms of engagement increase.

3.Minimax Regret Approach: First, you calculate the regret for each feature under each scenario. The regret is the difference between the best possible outcome and the actual outcome for each feature.

Scenario 1: Feature A is the best

Scenario 2: Feature B is the best

Scenario 3: Feature C is the best

Here, we are assuming that launching a feature B in a scenario where feature B is not best suited, will result in worst possibility of feature B

For example, in Scenario 1 if we launch feature B, we will get only 50 users.

Regret Table:

Next, you determine the maximum regret for each feature:

• Feature A: Max(0,60,40) = 60
• Feature B: Max(50,0,10) = 50
• Feature C: Max(60,40,0) = 60

With the minimax regret approach, you would prioritize Feature B, as it has the least maximum regret.

In conclusion, each decision-making approach prioritizes features differently based on their focus. As a product manager, you should carefully consider which approach aligns best with your business objectives and risk tolerance when making prioritization decisions under uncertainty.

6.3.2 Decision making in second class of uncertainty

As a product manager, when prioritizing features for development under risk (as opposed to uncertainty), you have a reasonable idea about the chances of each scenario occurring. In other words, you know the probabilities of occurrence for various scenarios that arise from implementing different features. Under this condition, it is easier to combine the payoffs (or outcomes) under different scenarios into a single number representing the effectiveness of implementing a specific feature.

Expected Value Approach:

Consider a set of features, each giving rise to a series of payoffs (or outcomes) corresponding to different scenarios that can unfold. If you, as a product manager, know the probability of the different scenarios, you can obtain a weighted sum of the payoffs from different scenarios in which each payoff is weighted by the probability of the corresponding scenario. This weighted sum is called the expected value of the distribution of payoffs.

For example, suppose you are considering Feature A and have four possible scenarios (I through IV) with probabilities of 0.4, 0.3, 0.1, and 0.2, respectively. The expected value associated with Feature A can be calculated as follows: 0.4×100 users+ 0.3×200 users + 0.1×150 users+ 0.2×500 users= 215 users

By performing similar calculations for other features, you can obtain their respective expected values. As a product manager, you should prioritize the features with the highest expected payoffs to maximize the overall effectiveness of your product development.

When making decisions under risk, the most common strategy used is maximizing expected payoffs (or minimizing expected costs for cost minimization problems). A part of the reason for doing so is that the expected value is one of the best-known measures for handling uncertain situations and is well-understood by all parties involved in decision-making.

6.4 Estimating Value of information

6.4.1 Expected Value of Perfect Information (EVPI):

As a product manager, when prioritizing features for development, you want to maximize the expected payoffs. Although you might have some idea about the outcomes of implementing different features, getting expert advice specific to your current situation can be valuable. In this context, the value of information comes from understanding how much the expert’s advice can improve your decision-making process and ultimately lead to better results.

Consider a perfect expert who is infallible in predicting the success of implementing a certain feature. How much is the advice of such an expert worth to the product manager?

The expert’s advice is valuable before deciding which feature to prioritize, as it helps the product manager make a more informed decision. Once the decision has been made, the expert’s advice has no practical use. Since the expert is always right, the product manager will not second-guess the advice and will use it as the basis for their decisions.

Assume that without the expert’s advice, the optimal strategy for the product manager is to prioritize Feature C, which yields an expected payoff of $1.19 million. If the expert advises that a different feature, Feature X, will be successful, the product manager can avoid implementing Feature C and instead focus on Feature X. By following the expert’s advice, the product manager might obtain an expected payoff of $1.25 million.

In this case, the value of the expert’s advice is the difference between the expected payoff with the expert’s advice and the expected payoff without it: $1.25 million — $1.19 million = $60,000. This figure is an expected payoff obtained by assuming perfect information from the expert and is called the Expected Value of Perfect Information (EVPI).

The EVPI helps the product manager understand the worth of the expert’s opinion in their decision-making process. By utilizing valuable advice from experts, product managers can make better-informed decisions when prioritizing features for development, ultimately leading to improved product outcomes.

6.4.2 Expected value of Sample information (EVSI)

Now, consider a situation where you have an imperfect expert advising on the success of implementing a certain feature. Imperfections in the expert’s advice can lead to two types of mistakes:

• The expert incorrectly claims that the feature implementation will be successful when it actually fails, causing the product manager to incur a monetary loss.
• The expert mistakenly states that the feature implementation will fail when it is actually successful. In this case, the product manager may choose to not prioritize the feature, resulting in opportunity losses.

Despite the potential for mistakes, an imperfect expert can still be valuable to a product manager. When the expert is correct, they can help the product manager avoid decisions that lead to losses. As long as the expert is correct more often than not, the product manager is better off with their advice in an expected value sense.

To compute the worth of an imperfect expert’s advice, you need to find the probability (p) with which the expert will claim that a feature implementation will be successful. Using the expert’s past decisions and applying Bayes’ rule can help determine this probability.

Suppose we have an imperfect expert whose past performance is as follows:

1. When the expert claims a feature implementation will be successful, they are correct 90% of the time.
2. When the expert claims a feature implementation will fail, they are correct 80% of the time.

Let’s assume that, in general, 70% of feature implementations are successful. We can use this information to calculate the probability (p) of the expert claiming a feature implementation will be successful using Bayes’ rule.

P(Expert claims success | Actual success) = 0.9

P(Expert claims failure | Actual failure) = 0.8

P(Actual success) = 0.7

P(Actual failure) = 1 — P(Actual success) = 0.3

Now, we can find the probability of the expert claiming success:

P(Expert claims success) = P(Expert claims success | Actual success) * P(Actual success) + P(Expert claims success | Actual failure) * P(Actual failure)

P(Expert claims success | Actual failure) = 1 — P(Expert claims failure | Actual failure) = 0.2

P(Expert claims success) = 0.9 * 0.7 + 0.2 * 0.3 = 0.63 + 0.06 = 0.69

Now that we have the probability of the expert claiming success (p = 0.69), we can create a decision tree for the product manager that includes the option of consulting the imperfect expert. By solving the decision tree, the product manager can determine the expected payoff with the expert’s advice.

Let’s assume the expected payoff without the expert’s advice is $1.19 million. After consulting the imperfect expert, the expected payoff increases to $1.25 million.

The worth of the expert’s opinion is the difference between the expected payoff with the expert’s advice and the expected payoff without it: $1.25 million — $1.19 million = $60,000. This represents the Expected Value of Sample Information (EVSI).

In this example, the product manager can use the EVSI to better assess the value of the imperfect expert’s advice and make more informed decisions when prioritizing features for development.

6.4 Best Practices for Managing Research Priorities in Different Organizational Contexts

Effectively managing research priorities depends on the specific context of the organization. In this section, we outline best practices for managing research priorities in various organizational contexts, providing examples for each.

Establish a formal research prioritization process: Create a structured process for evaluating and prioritizing research projects, ensuring transparency and consistency.

Example: A technology company may establish a research prioritization committee that includes representatives from product management, engineering, design, marketing, and sales. This committee meets quarterly to evaluate research proposals based on criteria such as alignment with strategic objectives, expected ROI, and resource requirements. The results of this evaluation are used to prioritize research projects for the upcoming quarter.

2. Encourage cross-functional collaboration: Involve stakeholders from different departments in the research prioritization process to ensure a holistic perspective and buy-in.

Example: An e-commerce company organizes a series of workshops where stakeholders from various departments come together to discuss research priorities, share insights, and identify potential synergies. This collaborative approach allows the company to identify high-impact research projects that address multiple needs and foster cross-departmental cooperation.

3. Adapt research prioritization to the organization’s stage and strategy: Startups may prioritize research activities that deliver short-term results and support rapid growth, while established companies may prioritize projects that deliver long-term value and innovation.

Example: A startup focused on developing a new mobile app might prioritize research activities that help them quickly acquire users and generate revenue, such as market research on user preferences and competitor analysis. In contrast, a large, established software company might prioritize research activities that drive long-term innovation and differentiation, such as exploratory research on emerging technologies and user experience improvements.

4. Continuously monitor and reassess research priorities: Regularly review research priorities and adjust them based on new information, market changes, and organizational goals.

Example: A healthcare company conducts monthly reviews of its research portfolio, evaluating the progress and impact of ongoing projects and assessing new research opportunities. This process allows the company to make data-driven adjustments to its research priorities, ensuring that resources are allocated to the most impactful projects and that the research portfolio remains aligned with the organization’s evolving goals.

By implementing these best practices, organizations can effectively manage research priorities in different contexts, maximizing the value of research investments and supporting the achievement of strategic objectives.

Conclusion

In conclusion, effectively managing research priorities and decision-making in product management requires a deep understanding of the fundamental principles that guide the process. By adhering to the three key principles mentioned in the chapter —

1. Principle of Optimized Resource Allocation: Balance research costs and expected benefits to maximize return on investment and ensure efficient use of resources in prioritization.
2. Principle of Contextual Decision-Making: Adapt decision-making approaches to the specific context and uncertainty class, aligning with organizational goals and risk tolerance.
3. Principle of Collaborative Adaptation: Foster cross-functional collaboration and continuous monitoring to maintain alignment with strategic objectives and effectively allocate resources to impactful projects

— product managers can ensure that their prioritization decisions lead to the efficient use of resources, align with organizational goals, and foster innovation and growth.

These principles serve as the foundation for making informed decisions under various levels of uncertainty and within different organizational contexts. By following these principles, product managers can not only maximize the return on investment in research but also create a collaborative environment that enables continuous adaptation and improvement.

Ultimately, the mastery of these fundamental principles equips one with the tools necessary to navigate the complex landscape of research prioritization and decision-making, supporting the achievement of strategic objectives and driving long-term success for their organizations.