Do these 3 Things Before You Take a Machine Learning Course

So you are a product manager who needs to learn fast to take on a machine learning project. When I started down this path about a year ago, I was tempted to join the engineers on my team in taking a machine learning course. The issue was 99% of the machine learning courses out were not designed for product managers. They were great for our engineers and data scientists to get the basics to get started, but they did not help me in my work.

Here’s what I suggest doing instead. In this article I will detail three steps specific to what product managers should do to get started with machine learning. For each step, I will give an example from my work at eSpark Learning and a prompt to help apply it to your project.

1. Define a valuable, actionable output

It may be tempting to dive into the technology to figure out what you need to do, but as a product manager you should focus on business value first.

While there are lots of more advanced approaches, the most common commercial implementations of machine learning boil down to building a system that will accurately predict outputs from inputs. The way it works is you feed the models enough existing examples of both inputs and outputs to learn from. Then the model can generate the outputs on its own. A classic application is spam classification. Gmail trained a model with millions of emails that were manually identified as spam by users. Now the system can take in new emails (input) and automatically mark messages as spam (output). So the most important place to start is defining what output will be valuable based on your inputs.

“Having a PM responsible for collecting a test set
is one of the most effective processes for letting the PM specify what they really care about.”

- Artificial Intelligence is the New Electricity by Stanford Professor, Andrew Ng

Example: In eSpark’s case we had a writing product for 4–8th grade students. We wanted to provide useful feedback on student writing to students, teachers and school leaders. Our product already used a four category rubric for peer and teacher review: Purpose, Organization, Evidence/Elaboration and Conventions. The problem was that generating scores on just these high level categories was not very helpful for our stakeholders. We needed the feedback to be specific and actionable.

This led us to develop a 12-point rubric with elements such as effective introduction, citation of sources and proper capitalization. We knew if we could accurately predict how students fared on those elements, we could provide actionable feedback. However, it was more like a discovery process than a delivery process. In the first few months, we tweaked the rubric several times as we learned about what would likely work from an engineering perspective but still be valuable to school partners.

A snippet of our spreadsheet that included inputs (grade level and Amazon link to student writing) and outputs, scores on a 0–2 scale for 12 elements of student writing.

Your turn: Try creating a spreadsheet with a subset of your data. In one column you have your input sources (e.g. text, video), in the other columns you have the outputs or predictions (e.g. classifications, scores) that you want your model to generate. This step does not require any existing machine learning models or expertise. It just requires domain and business expertise to identify what inputs you have available and what would be valuable information for your users. This set of data will facilitate a conversation with your product team about feasibility and value to highlight assumptions you may need to test.

2. Create a data strategy that generates value immediately

As a product leader you need to define a vision for how you will create a flywheel that grows your supply of structured data (inputs + outputs). This will make your products better over time. We quickly realized that collecting enough structured data was a harder problem to solve than finding the right machine learning model. Don’t go out and hire a machine learning expert or consultant until you have your data strategy in place.

“It’s good to plan for all the great things you can do with data collected in the future, but you have to offer some immediate value so that people stick around.”

- 12 tips for designing and managing an AI-driven product by Venture Beat author, Will Murphy

Example: In our case, the data strategy was to use professional human graders to score student writing. We immediately used those scores to generate reports for teachers and school administrators. We created an internal scoring tool tailored to our use case to speed up efficiency. Our vision is that over time more elements of the rubric are automated through machine learning allowing us to deliver feedback in real-time directly to students.

Our internal scoring tool which allowed us to efficiently score multiple linked elements that required access to reference materials (prompt, resources, rubric).

A report we generated for school administrators using professional graders to score student work on our 12-point rubric.

Your turn: Discuss and draft answers to these questions with your team:

1. How you will get the data now and the future? Will your users generate it in product or will you outsource it to a platform (e.g. Amazon mTurk, Figure Eight) or hire domain experts to do it?
2. How you will get value from it immediately?

3. Define the metrics for success

You need to guide your team’s focus on metrics just as you would for other team objectives. One issue with technical ML courses is you will get in the weeds learning to implement various models when you don’t really need to know those details. While it helps to have a high level knowledge of the models, your main focus should be on understanding the metrics to evaluate them and the quality of the data that is training them.

If you assume your system will make some mistakes, your job is to prioritize them. To use the spam example, is it better to miss some spam messages that show up in the Inbox or find all the spam messages but also send some real messages to the Spam folder? In addition, make sure you have data on the impact to your users because a great model doesn’t guarantee a great experience.

“Your ML system will make mistakes…While all errors are equal to an ML system, not all errors are equal to all people.”

- Human-Centered Machine Learning by Google Designers Josh Lovejoy and Jess Holbrook

Example: In our case we are just now getting enough data to apply some simple models to provide real-time feedback for students. In doing so we looked at a few metrics:

Data quality: A good metric for us was inter-grader agreement. We had multiple graders evaluate some of the student work to see how often they agreed. Then we followed up with the graders to clarify ambiguities or note issues.

Model quality: We focused early on precision over recall to build trust with students and teachers. This means some students will not get feedback that would have helped them, but we ensure the feedback we do give them is relevant.

User value: We track the rate by which students incorporate our feedback into their final drafts. This gives us a gauge of the impact that each piece of feedback has on student writing. To use a simple example, about a third of students who received our feedback to “Keep writing!” actually did write longer drafts!

Our real time Smart Supports for student writing prioritize precision and look for student changes from draft to final to evaluate performance.

Your turn: Read about the basics of machine learning metrics here. Then outline your own metrics:

1. Data quality: How will you audit your data?
2. Model quality: What are the goals for model performance?
3. User value: How will you know if it is working for users?

The world of machine learning is a fascinating one that will certainly transform the future. I hope you enjoy your journey of learning as much as I did.

If you are interested to learn more, here are a few opportunities:

• (Chicago) Let’s meet: Come to the Women in Product Chicago Machine Learning event on Sept 13th. I will be speaking with a few other amazing women ML experts.
• (SF) Take a course tailored to product managers: University of San Francisco has a new Data Science for Product Manager certificate program. Helen Mou, an amazing Shopify PM working on ML, is teaching the course. Andrew Ng’s online course also has some great videos.

Please let me know in the comments any additional information, questions or areas on your mind. I’ll consider them for future blog posts!