Why Product Managers, Designers & Data Scientists Add More Value as a Team

Photo by Kaleidico on Unsplash

Imagine for a second, you are a product manager of a product that helps doctors diagnose whether a patient “may” have cancer by looking at the photographic scans of the human tissue and then making a prediction about the presence / or absence of cancer.

Now clearly, doctors have been doing this manually, putting those scans against a light source and then using their experience and medical knowledge to make their prediction.

So, why does the world need this product? What is its value proposition?

1. Doctors get tired, and when they get tired they may make a mistake. These mistakes can be very expensive in human life and money.
2. Doctors have limited time. Any technology that may help a doctor use that time more effectively by focussing on the cases that are more likely to be serious, will be valued.

So, the problem statement is framed as “save doctors time by doing a preliminary diagnosis using our product, while maintaining a high rate of accuracy.”

(I am simplifying this quite a bit here, but I think it will help get my point across without diluting it.)

With that problem statement in mind, you approach your data scientist and say “I need a model that can accurately predict whether a person may have cancer by examining the photographic scans. I want that model to be at least 95% accurate.”

Unfortunately, this is where most product managers would leave the conversation with their data scientist.

That is because ‘an accurate model’ doesn’t mean much without further context and some decisions around how the results of the model would be acted upon (which is where the user experience designer comes into picture) and what would be the implications and costs of getting the prediction wrong, which is where the product manager must provide domain expertise.

In this article, I’ll try to shed some light on this.

Let’s get some basic principles right first.

To understand why that product manager’s ask is insufficient, we need to start by using terminology that has precise meaning and is used consistently by product managers, data scientists, and product designers.

The job of a data science model (in this scenario) is to make a prediction. In the use case above, let’s assume that the prediction is categorical, i.e. the model predicts by looking at the image whether the image indicates whether the tissue is “Cancerous” or “Not Cancerous”. There are four particular outcomes that can occur here (based on different combinations of what the model predicts and the reality).

Photo by Hasan Albari from Pexels

1. The model predicts that the tissue is Cancerous and in reality it is Cancerous. This is called a True Positive.
2. The model predicts that the tissue is Cancerous and in reality, it is Not Cancerous. This is called a False Positive.
3. The model predicts that the tissue is Not Cancerous and in reality it is Cancerous. This is called a False Negative.
4. The model predicts that the tissue is Not Cancerous and in reality, it is Not Cancerous. This is called a False Negative.

Anytime the model makes a prediction, any of these four outcomes could occur. When the model get’s it right, there is a benefit. When a model gets it wrong there is a cost.

Depending upon the type of business problem, the cost and benefits may vary.

To solve a problem effectively, the data scientist, product designer, and product manager together must maximize the benefit while reducing and mitigating the cost.

Going along with those four outcomes is a set of metrics that help quantify these costs and benefits.

Precision: Of all the scans that the model predicted as Cancerous, how often did it get right? This translates to “True Positive / (True Positive + False Positive)” or “Scans correctly identified as Cancerous / (Scans Correctly Identified as Cancerous + Scans Incorrectly Identified as Cancerous).”

For example, if the model predicted 10 scans as Cancerous, and 6 of them were Cancerous in reality and 4 of them were Not Cancerous, the model’s precision is 6/10 or 60%.

Recall: Of all the scans that are Cancerous in reality, how many of them did it predict as Cancerous. This translates to “True Positive / (True Positive + False Negative)” or “Scans correctly identified as Cancerous / (Scans correctly identified as Cancerous + Scans incorrectly identified as Not Cancerous).”

For example, if there were 8 Cancerous scans in reality and the model predicted 6 of them as being Cancerous the remaining 2 were predicted as not being cancerous (when in reality they were), the model’s recall is 6/8 or 75%.

Finally, you could calculate an accuracy metric as well. But in reality, as you’ll see below, it is not that useful.

Accuracy: how often did you get predictions (whether you are predicting a tissue as being Cancerous or Not Cancerous) right. This translates to “(True Positive + True Negatives) / (# of all the predictions).”

Reframing the question.

You may think that instead of asking for an “accurate” model, the product manager can ask for “a model that never predicts a false positive and never misses out predicting a real positive.”

If only it was that simple.

The reality is that any model will need to be tuned to either focus on ensuring that it never predicts a false positive (high precision) or it never misses out on a real positive (high recall).

The reason for that is the model learns from data and the model’s knowledge is limited to the data it is trained on. Since the trained data will always be a sample (based on practical and theoretical constraints), it will always provide a limited view of reality.

So the questions boil down to:

1. Do we want to optimize for never making a false prediction, even if we miss out on real cases?
2. Do we want to optimize for never missing out on a real case, even if we make a false positive prediction?
3. What training set do we have access to and to what extent is it a representative sample of the reality?
4. Finally, how do we do better than randomly guessing?

A couple of examples to make this even clearer:

1. You create a model that given a scan always predicts ‘Cancerous.’ In this situation, you will never miss out on any real case, but you will have lots of false positives (if only 1 out 1,000 people have cancer, you will be wrong 999 times). In that aspect, the model is quite useless.
2. You create a model that given a scan always predicts ‘Not Cancerous.’ In this situation, your model will never falsely call out a healthy person as cancerous, but it will miss out on all people who have cancer. Again, the model is entirely useless.

For this business problem, you would want a model that would ensure that anyone who has cancer is not omitted, while drastically limiting people who don’t have cancer from being flagged as having cancer.

Let’s bring all these concepts together.

In this final step, the team has to address the question: how would you design the experience of a doctor to interact with a system that is making these predictions:

1. What is the user experience we want? Based on the prediction of the model, would we ‘automatically start treatment?” Would we “automatically order new tests?” Would we “alert the doctor?”
2. For each of the possibly feasible user experience, we should ask some following questions: What is the cost of a false positive prediction? What is the cost of a false negative prediction? What is the benefit of a true positive prediction? What is the benefit of a true negative prediction?”

These questions are not answered linearly; rather you try to answer these questions iteratively. Because at the end of it all, your team is balancing the following three items:

1. The user experience designed around the people.
2. The economic and human costs and benefits of the various outcomes from your model.
3. The data set and the algorithms used to build the model.

My wrap up.

The main idea behind this article is to encourage a closer co-operation between data scientists, product managers, and product designers by shedding some light on the questions that as a team they should collectively answer. Also, what is the importance of these questions, as well as building some common vocabulary to facilitate deeper conversations within those teams.

If you are a product leader who can influence how teams are set up within your organization, I hope this article helps you think about how you set up team structures so that your product manager, data scientist, and designer are not set up in a silo but rather in a collaborative setting where they are asking and answering these questions together.

If you find this article helpful or you have follow-up questions or stories of your own to relate, I’d love to hear from you.