Machine Learning for Product Managers

Photo by Markus Spiske on Unsplash

Machine learning is the science of helping computers discover patterns and relationships in data instead of being manually programmed. It is a powerful tool for creating personalised and dynamic experiences, and it’s already driving everything from Facebook recommending who you should be friends with, Shazam finding the tracks you are listening, to autonomous vehicles and brain-computer interfaces. Undoubtedly, most of the products you enjoy on a daily basis use flavours of machine learning. But as more and more experiences are built with it, it’s clear that Product Managers still have a lot to learn about how to make users feel in control of the technology, and not the other way round.

As Mark Cuban stated:

“Artificial intelligence, deep learning, machine learning — whatever you’re doing if you don’t understand it — learn it. Because otherwise you’re going to be a dinosaur within 3 years.”

Machine learning can address different user needs, which are correlated with different mathematical problems. The majority of problems it can tackle, fall into the following categories:

Ranking

Facebook, deciding what news to add in your news feed, from the content posted by your friends to sponsored posts? Google, trying to answer your query with the best possible results? These are ranking problems — helping users find the right thing when they search.

Recommendation

Instagram, recommending new people to follow? Netflix, suggesting movies that you might be interested in? These are recommendation problems — giving users things they may be interested in, without them explicitly searching in certain occasions.

Regression

Hopper predicting the price of flight tickets? This is a regression problem — predicting the numerical value of a thing.

Classification

Facebook, detecting faces inside a photo you just posted? Gmail, marking an email as spam? These are classification problems — figuring out what kind of thing something is.


As a product manager of a machine learning feature or product, it is crucial that you understand machine learning. By that I certainly do not mean that you need to have in-depth knowledge of machine learning algorithms or that you should be able to come up with the actual models. Above all, product managers need to be familiar with the questions below:

  • How is machine learning going to add value to your product?
  • What can machine learning achieve for your product, and what would it take to execute it (more data, better algorithm, etc)?
  • Can you identify the difference between hype and real-achievable things from machine learning?
  • How different are the processes followed in building a machine learning product compared to a product without machine learning?

Trying to answer these questions, I have fallen myself or seen others fall into the following fallacies:

Fallacy #1: Focus more on the details of the model, than usability.

A lot of times data scientists build complicated models not needed by the business. Everything the data science team builds will be used by real people, and the fact that the algorithm has 12 features, 97% precision and 98% accuracy does not mean anything to the end-user. This is the reason why the rule of the MVP, with frequent iterations, also applies in such projects. For instance, a model might have 97% precision and that could be a huge internal success. However, putting it out there, that 3% might lead to bad user experience due to high volumes. Hence, online evaluation of the model matters as much as the offline does. On the contrary, a model that has 80% precision could seem low to the team and the stakeholders, but could deliver significant value to the business.

In addition, a higher number of features does not translate to higher usability. Features could be removed from the model with little or no loss in accuracy, while maintaining them could lead to the model assigning them some weight, with potential negative impact in case of future changes. Therefore, it should become clear that methods such as A/B testing, frequent releases, and MVP are crucial for the success of machine learning projects.

Fallacy #2: The business metrics and the mathematical metrics do not need to be aligned.

Machine learning models that do not align with the business models not only waste time and money for no reason, but also undermine the organisation’s trust in machine learning. This is especially true in companies that are late to data science, have skepticism about the power of data science, or are very qualitative in their leadership.

The success of such a project lies on the initial business questions and business metrics that should be addressed through machine learning. In fact, these answers will drive the data science team to decide what types of algorithms they will use and what metrics they will follow. For instance, at the beginning of every project, GitHub’s machine learning team defines not just the problem or question it addresses, but what the success metrics should be.

In addition, in a ranking problem data scientists usually check the precision@k metric, but what should be answered first of all is how this mathematical problem is linked with the business problem e.g. increase the clickthrough rate of the user.

Fallacy #3: The same model can solve different problems, since they seem similar.

One of the easiest traps in machine learning is to believe that the same model can be introduced in different components, because they address similar problems. Nevertheless, there is no guarantee that the model will be useful if we moved to a new context. For example, at Workable we had to create two separate identity resolution models, as although the task of both components was to find which pairs of candidates refer to the same person, the input data was slightly different in each case.

Remember, these algorithms draw their power from being able to compare new cases to a large dataset of similar cases from the past. Thus, when you try to apply a model in a different context, the cases in the dataset might be different now, and what was a strength in the original context is now a liability. However, there are cases where an out-of-context model can still be an improvement over no model at all, as long as its limitations are taken into consideration.


By all means, transitioning to a product that follows different principles is definitely challenging. As Fast Forward Labs’ Friederike Schuur claims:

It is difficult for people to wrap their heads around what it means to work within a world where everything is just a probability. People want exact numbers and definite results. Even just two days ago, I was in a meeting and we had built a proof-of-concept prototype for the client, and they just went, ‘Does it work? Yes or no.’

As more organisations aim to instil machine learning inside their products, it is necessary for product managers to embrace and lead the behavioural change generated by the introduction of this powerful toolkit for building complex systems quickly (if done properly).