Identifying Opportunities for Using Machine Learning as a Product Manager

Frameworks and examples from 15 product experts at leading technology companies

Background

This article is part of a larger independent study (see posts below) about how product managers are incorporating machine learning into their products. It was conducted by Ryan Dingler and myself while MBAs at UC Berkeley with the help of Vince Law as our faculty advisor.

The research is an attempt to understand how product managers are designing, planning, and building machine learning enabled Products. To develop this understanding, we interviewed 15 product development experts at various technology companies. Of the 15 companies represented, 14 have a market capitalization greater than $1 billion, 11 are publicly traded, 6 are B2C, and 9 are B2B.

The product managers guide to ML series:

• Identifying Opportunities for Using Machine Learning as a Product Manager
• 4 Steps for Product Managers to Get Started with Machine Learning
• Creating a Data Strategy for Machine Learning as a Product Manager
• Principles for Product Managers on How to Manage a Machine Learning Model

Identifying opportunities

If you’ve worked on a product team in the last few years, you’ve probably heard someone (likely the PM) ask,

“Can’t we just use machine learning (ML) to solve that?”

There’s a common notion that ML can solve a lot of the challenges that product managers face. With virtual assistants on our phones and personalized recommendations after every purchase, it’s hard to deny that ML is changing the way products are built and consumed. However, it’s often difficult to know where to leverage ML in your product.

Why does it matter?

When it comes to machine learning, finding what problem to solve next is crucial. Resources like data scientists and ML engineers are limited. Selecting the wrong project for your team is not only costly, but it can also hurt morale, customer trust, and lead to failed products.

Proper problem identification can help prevent your work from ending up in the graveyard of ML models and products.

What problems can machine learning solve?

In our research, we came across a large variety of ways companies were using ML in their products. However, we noticed some common trends among all these use cases and distilled them into four (sometimes overlapping) problem areas: detecting anomalies, filtering information, moderating content, and automating repetitive tasks.

These areas are meant to provide examples of what types of business problems might be good candidates for ML.

Note: We included some examples that were similar to those we came across in our research. The identified areas are ones we directly observed during our interviews and are not a complete list.

Detecting anomalies

ML is excellent at detecting patterns in data. This strength can be leveraged to help users more easily find data points that don’t match the typical pattern. In our research, we found companies that applied supervised learning, unsupervised learning, or even a combination of both for anomaly detection. The approach chosen depends on the use case.

Examples

Supervised: Companies regularly audit their accounting books by spot-checking for fraud or mistakes in hundreds of millions of entries. These audits are often very manually due to the technical domain knowledge required for this task. Years of these manual audit results produce excellent labeled data for supervised models. These models produce outputs that are easy to interpret and explain. That said, the models can miss new types of fraud or mistakes that were not similar to the training set.

Unsupervised: In digital advertising, given no controls, there is a significant incentive for publishers to click on ads on their properties and for companies to click on their competitors’ ads. Simple heuristics can prevent some fraud, but unsupervised models can find new patterns in the data, making them excellent at detecting different types of fraud. Based on data such as the user IP address, transactions, and timing, unsupervised models can identify bad actors. However, these models often produce predictions that are opaque and difficult to interpret.

Combined Approach: Supervised and unsupervised learning is often the most effective approach to anomaly detection. Let’s say an unsupervised model found fifty users that it thinks are fraudulent. A supervised model could be applied to provide more details on why these users might be frauds (e.g., IP addresses are the same, similar timestamps, etc.). Other methods, like semi-supervised, may also increase performance.

Filtering information

Users are often overwhelmed by products with too much information. There are two basic approaches to solving this problem with ML: search and recommendations.

Search

Search is when a user is trying to ‘pull’ information. Sometimes, users need to find information or objects but don’t know what to look for or where to find them. A simple search algorithm could use text matching and recently viewed items to look for the object, but ML can do much more. ML models can consider hundreds or thousands of features in search results in a way that rule-based search can’t.

Examples

Search is core to Dropbox’s experience. When a user searches for “Machine Learning” in their organization’s Dropbox, a collection of documents are returned and then ranked. The resulting ranking is based on more than the query text (“Machine Learning”); it also uses a relevance score. This score takes into account the searcher, who they interact with, and what files they recently opened (freshness). A model like this can be trained using readily accessible data like past user searches and click results.

With increased access to massive computation, image, video, and audio search have become possible. Even without manual labels for Facebook or YouTube videos, ML models can extract audio and use image recognition to index the videos for search. Similarly, Squarespace uses visual search to help its users find sites with similar home page images.

Recommendations

If Search is a ‘pull,’ recommendations are a ‘push.’ Like in search, recommendation ML models help users navigate information overload, but recommendations do so by pushing personalized information to the user. The most common application of recommendation ML is in social media newsfeeds and Amazon’s “customers who bought this item also bought.” However, other products are starting to push personalized recommendations to their users as well.

Examples

When a user opens Instagram, Reddit, or LinkedIn, ML models automatically serve up a personalized and mindless experience filled with updates on people or subjects of interest. Even ads can be so embedded and personalized that they become part of the recommendations experience.

Open the Nordstroms shopping app, and the first thing you’ll see is a curated set of products “for you.” It uses ML to help customers discover sales and new products that they might like based on prior shopping history. Nordstroms also helps you find products that you haven’t viewed or bought before, but that others like you have purchased. Many retailers now leverage ML for recommendations in their online presence.

Moderating content

More and more companies are relying on user-generated content in their products and moderating that content is becoming increasingly important. Content like the photos, text, audio, videos, or even live streams needs to be moderated to abide by the rules set by each platform. It’s impossible for companies to have human moderators enforcing these rules for all of their content.

Examples

YouTube has 500 hours of video uploaded every minute, which would require more than 100,000 reviewers working 40 hours a week. Instead, businesses like YouTube rely on users and ML to flag content for a moderator to review. In this use case, ML is supposed to mirror what a human reviewer would do because the data is all human-labeled, and the success metric is passing human review. This reliance on human judgment is what makes moderating content with ML continue to be a fundamentally challenging problem to solve.

Reddit, on the other hand, does very little moderation on its platform. It has optimized for freedom of speech, so the platform is mostly moderating illegal content. On the other side, professional networks like LinkedIn consider any inappropriate content to be a huge issue. Platforms like LinkedIn moderate content with ML models tuned for the cleanliness of the platform over upsetting a group or individual.

Automating repetitive tasks

The final problem area where we saw many ML applications is in automating repetitive tasks. These are tasks like predicting the quality of sales leads, inputting and categorizing receipt data, or sending marketing emails. This area is often an excellent place to start in ML because labeled data is abundant, and there is a direct time-saving.

Examples

Submitting receipts for expense reports is a very repetitive task, being automated with optical character recognition and ML. The ML models take images of receipts and automatically fill in fields in the user’s expense report. Old, manual reports provide the information needed to train this model, making the data readily accessible (although SaaS vendors often have to ask for access to customer data).

Even complex tasks like email writing can be partially automated. Gmail’s smart compose uses previous word sequences entered by users to predict the next words in a sentence.

The model also uses the email title and any previous emails in the string as inputs. This approach takes the parts of email writing that are repetitive and manual and automates them with ML.

Final thoughts

If you have a problem that fits into one of these four areas (or another one not covered), consider reading Produce Manager to-dos when Starting a Machine Learning Project

ML techniques and capabilities are always changing. The problem areas laid out in this article are not meant to be comprehensive, nor are we implying that all problems can or should be solved with ML.