What does it take to build a product with machine learning?

In April and May this year, we hosted two meetups (this and this) that outlined considerations in building products based on artificial intelligence (and machine learning) from a product management point of view.

Early bird guests prepare diligently for the meetup talks

The talks in the meetups covered key points product managers (or business managers) should take into account when considering adding machine learning into their products:

1. Understand the basic premise of machine learning

Artificial intelligence (AI) became popular in the 1980s (Wikipedia), but its recent boom is mostly due to machine learning: the idea that one can use historical data to predict future behavior.

In a nutshell, the main concepts in machine learning are:

Here we talk about the basic concepts of machine learning

• There’s some data that you have. For example, say you have photos of cats and dogs, each one “labeled” with the animal shown in the photo.
• You run a training procedure. In our example, the training procedure trains the machine to recognize the difference between cats and dogs.
• The output of the training procedure is a so-called model. This is machine learning jargon that denotes what the machine learned.
• Using the model, the machine can now predict, for new unseen data (say a cat photo), whether it’s a cat or a dog.
• Most machine learning can be described as prediction (the machine tries to identify which class something belongs to — in our example, cat or dog) or regression (an awful term in machine learning jargon that denotes prediction of a numeric value, such as the price of a house, given data that describes it).

One example in the Gong world is a task called punctuation restoration. Historically, speech-to-text systems convert audio into words, but do not include punctuation (periods, commas, and the like). The punctuation restoration task is one that tries to tell (predict), after each word, which punctuation mark is most appropriate, as shown below:

A Gong sub-system converts sequences of words into readable text using machine learning

2. Assess whether machine learning is applicable to your product

The first thing to consider is what interesting predictions your product can make to help its users. You may want to ask yourself:

• Are these predictions valuable?
• Are they worth the effort?
• Do we have (historical) data to teach a machine learning system?
• Like every new product functionality, how can we validate the need and value? Also, is there a way to assess the accuracy of the system to get a sense of the potential value?
• Is it worth the cost and effort vis-à-vis other new product functionality?

Here’s what Andrew Ng has to say about the feasibility of machine learning nowadays:

3. Define the scope of the machine learning process

We found out that there are four common types of machine learning processes when applying machine learning to business applications:

• Internal use: In this case, a company builds a model that helps the system behave better behind the scenes. A common example is ad tech: in this space, a bidding algorithm uses machine learning to improve bidding on ad real estate, unbeknown to (almost) any user.
• Global use: In this case, a company builds a model that works across all of its customers and users. The punctuation example shown above is a good example. Punctuation rules and semantics are agnostic to the customer and domain; hence once a system is trained to restore punctuation, it can do so successfully for all customers.
• Customer-specific: In this case, a company builds a specific model for each one of its (business) customers. This allows better tuning, but raises several questions:
  – How much data is needed until a customer-specific model can be built?
  – As a business, how do you demonstrate value early in your sales cycle?
  – Can you leverage a global model until a customer-specific model can be built? (This relates to the machine learning term transfer learning.)
  – What’s the process you use to decide when to train such model, when to update it? Also, how do you measure the quality of such models (aka performance)?
• User-specific: In this case, a company builds a specific model for each individual. This allows even better tuning, but obviously makes the above questions even harder to answer.

Here’s how the Google keyboard “predicts” that I need to speak with my colleague, Eran Aloni

4. Understand what it takes

As a product manager or a business owner, you are most likely not an expert in machine learning. So in venturing into machine learning, you will probably need the assistance of such experts — people who can formulate the problem more formally, help collect and cleanse the data, try out different approaches, measure the accuracy, and possibly end up implementing the best approach.

While the name varies, in most cases, experts in this discipline are called data scientists, though names such as researchers, machine learning experts, and others may also come up.

You should consider whether in-house experts are needed (typically when there are ongoing machine learning innovations) or outsourced help is sufficient.

You should also realize that adding machine learning raises several other considerations:

• Nowadays Python is the lingua franca of machine learning. Unless your system happens to be implemented in Python, you’ll incur engineering and/or DevOps costs for this new technical environment.
• You will need to implement new technical processes for collecting the data, cleansing it, evaluating results, and so on. These processes are unlikely to go away after the initial version.
• You may need a process for “labeling”: tagging data with the “right” label so that the machine learning algorithms can learn. The process varies drastically between global models (trained once) and customer-specific models (retrained for each new customer). Though more rare, some machine learning approaches use unsupervised learning, in which the machine learns without any labels.

All in all, you should expect collaboration between product people, data scientists, and engineers to come up with a working system.

5. Plan how to present machine learning to end users

If you’re building a GPS system, the interface with end users is quite simple: “Turn right.” But if you’re building a more complex system for businesses, the presentation with and communication to business users is much more challenging. Some considerations include:

• Are the specific business users open to getting feedback or advice from a machine learning-based system?
• How do you communicate it to end users? Specifically, how do you avoid losing your users’ trust when the predictions are off?

Gong coaches sales professionals whose talk time in certain types of calls exceeds best practices. We chose to highlight potential improvement opportunities using yellow (vs. red) to make it clearer that the advice may not apply in each particular case.

• What type of control do you give to customers’ business owners and administrators over what’s presented and over the process of training the system? Can they fine-tune the recommendations? Turn some off?

In some areas of the Gong product, business owners can turn recommendations off to avoid presenting data that is not aligned with business owners’ priorities

Some Final Thoughts

I believe that the hype around AI blurs the picture and conceals some important truths.

One such truth is that AI gradually rolls out computer systems that help people do their jobs better and differently. In business settings, this means that AI systems are a combination of an assistant (“Computer, do this for me”), a coach (“Human, consider doing this next time”), and a manager (“Human, do this now to succeed”). (If the last point sounds scary, think about the role of GPS the last time you drove.)

Such systems will gradually be rolled out by industry (e.g., agriculture) and by role (e.g., sales professionals). As the press likes to remind us, in some cases (e.g., drivers), it may even replace roles altogether.

Understanding the considerations and process for building products based on machine learning is key to being part of this game, so my hope is that if you’ve missed our meetups, this post is of some help to you.