The Power of Product Scoring Mechanisms

One of the biggest challenges that both entrepreneurs and Product Leaders face is prioritization. How do you choose what to build now? What is more important? More urgent? Especially when you have a big vision and start to scale it, it becomes harder and harder to prioritize.

What do we need to prioritize?

It is not uncommon to hear from startup founders or product managers that they are extremely busy, and their task queues are getting longer and longer. I saw a month ago the to-do list of a product leader I know. I think there were more than 30 tasks on that list with only 3 done.

Because our job is so comprehensive and includes design, engineering, marketing, business, legal and more we need to prioritize almost… EVERYTHING! We do not have to prioritize only product features within our product roadmap, but we have to prioritize who to hire, our work-life balance and the time that we spend with the team and customers.

What is the problem with prioritization?

The main problem that entrepreneurs and PMs have with prioritization is that they are biased creatures. In fact, all humans are. Startup founders are in love with their idea and convince themselves that this is the right and best solution for the problems they are trying to solve.

Product leaders do the same as well. They assume they know their customer and have an opinion and intuition about what their customer want. While great customers and product people do know their problem space and userbase, being honest with yourself and your team is important and like in any experience, raise a hypothesis and back it up with data.

The question is how do we avoid being biased when we come up with a hypothesis?

What is Product Scoring Mechanism?

Product Scoring Mechanisms (PSM), are mechanisms that we build to help us make better and more objective decisions to fight our own bias. I’m a mechanical engineer by training (although I never worked as one), but if there’s one thing I learned at University is how mechanisms work.

A very simplified way to describe a mechanism is a system that takes an input and deliver an output. What happens inside is how you design the mechanism.

How a Product Scoring Mechanism looks like ?

The illustration above is a very simplified version of a PSM, however it is a very powerful one as well.

How to design a Product Scoring Mechanism?

Designing a PSM includes several steps and we can organize them all in a table:

1. Features

For this example, I chose to prioritize product features, but entrepreneurs, product leaders, and almost anyone can prioritize with the PSM method anything that he or she would like: tasks, potential hires and more. Let’s assume that I’m building Airbnb in their early stages. Choosing your features could come from a team discussion, work with your stakeholders, customer feedback, etc. Here is a list of the features I want to prioritize in the PSM table on the left column.

You do not need to prioritize the features in this stage since this is what you will do after scoring the PSM table.

2. Setting parameters

These are the parameters that you and your team think will affect your product prioritization. For example, if you are planning to build 3 product features with your team, each will have a score for the 3 different parameters that you defined. Examples of those parameters could be time to build, complexity, cost, etc. Here is an example where you would place the PSM parameters in your table (top row of the PSM table):

3. Setting weights

Because not all apples are red, each parameter could have a different weight and impact on your total decision. Maybe you are short in funding and cost would have the larger focus, or you are in fierce competition with 2 other startups and then Time to Build would be the most important parameter. There are many ways you can define PSM weights. The most common way to define weights is to choose a number that could be from 1–10 or 0–1 and then multiply each score with the weights that you define. To make it clear to your team and stakeholders, mark the weight next to the parameter in the top row of your PSM table:

4. Setting a scale

Setting a scale for your parameters could be done globally for all parameters or individually for each. Setting a scale is a tool to help you align everyone on how they should score the different parameters. What is the minimum score value, maximum score value and what each one mean. For example, let’s set the scale for the 3 parameters we defined:

1. Time to Build (1) — [(1)-Within a week, (2)- 3–4 months, (3) — a year]
2. Complexity — [(1)-Simple to build, (2)-medium complexity to build, (3)- hard to build]
3. Cost in $US — [(1)-1,500–10,000, (2)-10,001–100,000, (3)-100,001–300,000, (4)-300,001–500,000, (5)-500,001–1,000,000]

You can observe that we set the scales for each parameter in the PSM. In some occasions we can make the scale more specific and down to numbers like in the Cost scale, and sometimes it is a more ambiguous and loose scale like the complexity. It is all a matter of how granular you want your scale and PSM to be.

Another thing you can notice is that the scales are not equal. The Time to Build and Complexity go from 1 to 3 and the Cost from 1 to 5. For simplicity you should aim to create a solid and uniform set of scales, however, for the sake of the example, I’ll show how we can deal with different set of scales by normalizing them.

5. Scoring (input)

After you clearly defined the PSM scales and aligned the team, scoring is the actual work for you and your team to think objectively on each parameter and evaluate it for each feature. It is important that each team member will do this on his own to prevent influence from other team members. However, if you feel that a collaborative work would be better you could try it — you know your team best. Here is an example of the PSM table with scores of one of the team members:

Adding the weights which appear next to each parameter we will get the next weighted PSM table. We multiply each score by the weight we defined for the specific parameter.

6. Normalization (optional)

Normalizing your scores is optional. If you keep your scales uninformed, there is no need to normalize them. However, if you do not keep them uninformed or use weights in your PSM, you would want to normalize your scores. In order to normalize your scores we will do it in 3 stages:

a. Defining minimum and Maximum values of our original scales: In our case we are talking about the weighted values.

A=1 = 1×1, B=15=5×3

b. Define maximum and minimum values of your intended normalized scale: Let say we would like all the data to be from 1 to 5.

a=1, b=3

c. Compute all scores according to the normalization conversion formula: where x is any of the scores before normatlization.

Here is the normalized PSM table:

If you are good with not normalizing and dealing with multiple scales, that’s fine. However, leading a product is mainly about alignment and clarity so — normalize!

7. Output (Total Score)

Remember our mechanism and that it compute the numbers that we put inside? It’s time to compute the output. Like every part of our Product Scoring Mechanism it depends on how we build it, but for simplicity, we will compute a total score for each feature. I will use a simple summation of the different parameters to get one score for each feature. Here is the PSM table after computing the total scores:

You can see that the scale is different as we normalized it to be in the range of 1 to 3. So we entered 3 values as inputs for each feature — overall 12 values as input and we got 4 values as output.

8. Prioritization

Now it’s time for prioritization. This is the answer to the question that you asked. The question is: are you looking for a minimization problem or maximization problem? In our case, a good solution would be something which is quick to build, simple and doesn’t cost a lot. Parameters that could create conflict are importance or value to customer, but then you would have to adapt your scale for minimizing or maximizing.

Because we’re looking for a minimum I’ll start from the lower value to the higher one. In our example, it’s quite simple because the results are already ordered by ascending order. Here are the features our PSM mechanism got: Follow apartment button -> Cover photo for profile -> Adding map to apartment -> Adding second verification.

9. Benchmark

In some cases, you would like to set a benchmark or a default score in your company or team. This benchmark could be set after trial and error to validate in the same scale what would be a score that beyond that score you would decide to launch for example and below to archive a feature. A benchmark could also be a score that you would retrieve from market research or data analytics. The most important thing to remember is to keep the benchmark and your PSM in the same scale, or else it means absolutely nothing.

10. Recommendations

Now your product recommendation can have an objective an unbiased back. You will be able to confront your intuition, emotions, and ego with numbers and not only your numbers but with your team’s PSMs. However, making product decision will be much easier when you have a mechanism that helps you make your decisions slightly better and clear.

What is the problem with the Product Scoring Mechanism?

We’re back again with bias. Critics would say that the way you build your Product Scoring Mechanism is biased, and therefore the scoring and output are still biased. I agree — we can manipulate any mechanism to give the results we wished for. However, when dividing larger problems to smaller ones as we do in this method we minimize our bias into problems that we can quantify more clearly, and sometimes really measure a number like cost or time to develop. Building your Product Scoring Mechanism with a diverse team will also minimize the bias of this mechanism.

Back to Mechanical Engineering

If we imagine a real cog-wheel mechanism we put some data inside. Each cog wheel represents a parameter as we saw above and the radius size of each cog wheel represents the weight of each parameter. When they are in motion we are computing the total scores. What comes out from this mechanism from the other side is the output — a value.

This value will change according to how you design your mechanism and your input. Think of a car. You hit the gas, drive on a bumpy road and you drive an old car — it all will affect your output which is your car velocity. You will drive slowly and get home in 3–4 hours. If you buy a new car, drive on a flat and new road and you hit the gas with the same amount of fuel injected to the engine, you’ll get a different velocity, and you’ll get back home faster. You need to design your Product Scoring Mechanism as if you designed a car and you should be launching great products.

What are your thoughts about the Product Scoring Mechanism (PSM)? Would you use it for your product prioritization? for your startup? Let us know!

Meet the Author:
Illai Gescheit

Illai Gescheit is currently helping entrepreneurs and startups find their product-market fit. He led the Startup Programs and Products at Amazon Web Services in EMEA. In 2014, he founded Mobifile and also developed Patent Thinking. Illai is constantly contributing to the Product community onMedium.

*This article was originally published on the Product School blog.