Machine Learning is a Hammer.

Is your product a nail?

Silicon Valley loves to mock itself. We live for the latest Halting Problem writeup or Sara Cooper tweet. Every shiny new technology — chatbots, blockchain, AR/VR — gets its moment in the spotlight before the (limited) use cases are revealed… and it fades into obscurity again, appearing only in satire about our optimism.

But I’ve noticed a blind spot.

Ready?

Machine learning.

Before you get up in arms, let me explain.

At no time have I ever been called into a meeting to discuss how blockchain or AR could revolutionize my product. (OK maybe, in the context of a hackathon, with the explicit acknowledgement that the business would never support such a thing). We know that it is poor product management to start with a solution — especially a vague category of techniques, which can hardly be called a solution — and go looking for a problem to solve. When we do this in other contexts (how could we use chatbots to accelerate our business?) we are all but laughed out of the room.

But for some reason, we don’t feel this way about machine learning.

It is not unusual at all for executives to convene a task force to “find ways to apply machine learning in our business.” Or to send employees out to conferences, consultants, or collaborators to try and find a way to make this secret sauce their own. They seek an application instead of pursuing a problem.

Why is that? Why do we feel this obligation to one Silicon Valley buzz-term over others?

I have a few hypotheses:

1. People mistake the success of FAANG as fundamentally created by machine learning, rather than acknowledging that ML is an accelerator on top of a pre-existing money-making engine.

There’s no denying that machine learning has played a significant role in making these companies successful. But it’s a role enabled by a strong core product with a clear funnel of user value to optimize. Matching people to ads, to the right products to buy (fundamentally ads), to the right movie previews (also fundamentally ads), and, wait, to ads again is a great job for machine learning.

But the ML-driven matching is supplementary to what makes FAANG products great: sticky social networking; fast and convenient shipping, with access to products people want to buy; a set of compelling movies and TV shows that people want to watch. Without these core product investments, there would be no role for machine learning. There would be nothing to optimize.

Companies are often clamoring for investments in machine learning, hoping that either the technology or the marketing message will save them. But what’s truly needed is core product investments… that solve a real problem… in a way that customers will pay for.

2. Real differentiation is hard — teams are looking for a halo effect where they can get it. Machine learning is an easy grab.

We understand. You need to stand out. Because machine learning is associated with success, and most people barely understand it anyway, you can slap it into your marketing materials for all (attention) gain and no pain…

…until you waste time and resources trying to figure out what that marketing message really meant. And then build something to try and fulfill the words on the page, even if you don’t fulfill the promise. And you edge out the more valuable work you could have done with the same resources.

3. It’s easy to have a shallow understanding of machine learning that misleads about its true capabilities.

With other categories of high-profile tech buzz — AR/VR, chatbots — it only takes a couple of interactions with real use cases to realize that the technology just doesn’t work as well as the hype.

Because machine learning is “in the background” optimizing a metric, it’s hard to say from a demo whether the application provides a meaningfully improved experience. Seeing the right video or product or friend recommendation could be a fluke, or a handcrafted human choice, or a really good rules-based heuristic. The success of ML is often evaluated using high-level business metrics (look at all the time people spend watching Netflix!), and it’s difficult to parse apart how much machine learning really contributed (vs. the content itself, the new UX design, a marketing campaign running when you launched your latest model…). If you’re optimistic, or you have a shallow understanding of the tech, it’s easy to assume an outsized contribution.

This allows for a lot more fanciful storytelling around machine learning than around other popular technologies. Vague references to the outputs of a machine learning model and the features used enable businesspeople to fill in the blanks with their imagination… and unfortunately, what they imagine is often far more sophisticated than what machine learning can actually do with the data that they have available.

If all of this is true, the solution to our problem is to create a layman’s approach to understanding if your team actually needs machine learning. My attempt is as follows:

What you actually need for a machine learning investment to make sense for your team:

• A substantial amount of relevant data — a large data set to start out with, with a path to rapid growth over time. Clean, structured data in a format that your data science team can access and build models upon. Data that actually has a believable relationship with a behavior or outcome you want to predict.
• A large inventory of stuff to recommend or respond to, with the ability to add more over time — Amazon or Netflix are made great largely by their massive libraries of items to offer you, and by how those libraries change over time — if they only had 5 options, machine learning which one you’d like most wouldn’t do them much good.
• A use case where small, continuous amounts of incremental lift actually drive business value — Yes, ads fit this use case pretty well. You have to define critical actions or funnels and understand what you want to optimize for. Machine learning isn’t going to figure out your business goals for you.
• A use case where false positives don’t create substantial harms — ML models are essentially running many tiny experiments, some of which will fail. Seeing an unappealing ad on Facebook is unlikely to make you delete your account, or stop chatting with your friends. But using ML to block registrations for your service, or to dole out discounts, can have substantial negative consequences without the right guardrails in place.
• A use case that is relatively permanent — ML models need regular investment and maintenance, just like any other software. You’re unlikely to reap rewards over time if you’re treating ML as a “one and done” investment.

What most companies have:

• A small amount of data. A dataset so small that you can barely run multiple A/B tests, let alone other experiments. Data that is not clean or accessible by the relevant teams, and no tooling to work with the data in an appropriate way. Even better, maybe the data has only a weak correlation with behaviors you want to predict.
• Limited resources that aren’t being spent on creating a new inventory of assets to recommend (i.e. there aren’t enough options, and the options aren’t different enough, for machine learning recommendations to do much good).
• Poorly defined use cases that look good in headlines or marketing materials, but do very little to advance the company’s fundamental value or differentiation.
• Use cases where the ramifications of false positives have not been evaluated…
• … And the need for maintenance over time has not been scoped.

For the sake of clarity, this is not a hit piece on the value of machine learning or on the many extremely talented data science and engineering teams who enable that value. We need look no further than Silicon Valley darlings to see the impact that machine learning can have on a business.

However — this is a call-to-action for product managers and other business leaders to take off their rose-colored glasses and acknowledge that ML is just a technique, like many others, that is expensive to get up and running and should be utilized strategically. If you want to do exploratory R&D work with no line of sight to revenue improvement this year, that’s also fine. But call it what it is. And don’t use it as a substitute for doing your homework to understand where machine learning could make the biggest impact on your business.

It is very possible — likely, even — that to hit your numbers this year, you do not need machine learning. And that the increased efficiency of putting your dollars toward higher impact initiatives could do more for you than any bemused data scientist, looking at your 10,000 rows of user data, ever could.