Data Design Sprint — A Guide to Building AI That Works

Virtually all of the Big Tech companies — Microsoft, Google, and Facebook have invested millions of dollars and hired top talent to build AI solutions. Yet, many of them have ended in disaster:

• Microsoft’s AI, Tay, was a chatbot deployed on Twitter to have playful and casual conversations with its followers. Instead, it began to take on the form of a raging bigot, spewing racist and misogynistic tweets.
• Google’s Panorama feature promised to stitch photos together to create one continuous image. Instead, it would superimpose people into landmarks, creating wonderfully bad photos.
• Facebook created a chatbot that they had to quickly dismantle because they soon began to speak their own language

Each of these models made poor decisions not based on the data they had, but based on the data they didn’t have.

The three most important aspects of training a machine learning model can be broken down into:

1. Input Data
   Given the prediction you would like to make. What features, or inputs are correlated with that outcome? For example, if you’re predicting product sales, you can imagine that inputs like popularity, brand awareness and distribution channels would be positively correlated with sales.
2. Training Data
   Given the input data you’d like to build upon. Where do you procure this data? How do you ensure that the data are cleaned and representative of the population you’re interested in?
3. Feedback Data
   How do you build a model such that the predictions it makes then provides feedback to improve your algorithm?

Photo by Jo Szczepanska on Unsplash

I created the Data Design Sprint to compress months of team debate into a single hour of brainstorming and planning.

The process borrows from the Google Design Sprint Methodology (Jake Knapp) and adapts it for thinking about building AI that works. Here’s what you’ll need:

1. A 1-hour block of time where all stakeholders are in a room, focused. Aim for three types of Stakeholders, those who understand the business’ goal, technology requirements and user needs. Keep the meeting to a max of 6 people.
2. Each member of the team will work individually, and then pool their thoughts together during the brainstorm. Each member is capturing their thoughts using a sticky note. One thought per note.
3. The team converges to discuss each sticky note in order to prioritize next steps

The Data Design Sprint is designed to help your team align on the most important outcomes for building a machine learning model

The process is as follows. Note the times next to each section. You’ll need to be strict about the limits. Trust me, your team members may not believe that they can do this, but a little encouragement goes along way.

1. Define the prediction problem (20m)
2. Brainstorm the inputs and theme (10m)
3. Brainstorm possible bias outcomes of each input feature (10m)
4. Summarize, Capture & Share (10m)

At the end of this process, you will have created a product/feature strategy that allows you to test multiple ways to approach building your machine learning model.

At the conclusion of your sprint, you will be ready to experiment. Your team will have enough information to align and execute on a hypothesis.

Building successful machine learning models is extremely complicated and filled with unforeseen outcomes. Only by experimentation and monitoring will you be successful.

*Due to COVID-19, I’ve been running these sprints using Trello and other synchronous team tools.

1. Defining the Prediction Problem (20m)

Photo by Sam Chow

Goal: To align the team on the problem

In the first 15 minutes, ask your stakeholders to discuss the business problem and goals amongst themselves.

While the stakeholders speak, the team will be busy creating stickies on what they think the prediction problem may be. For instance, during the discussion the stakeholders may mention that one of their goals is to increase the average volume of items purchased for the next quarter.

A data scientist on your team may write: “Predict an item user may want” or your engineer: “Predict average user budget for specified month”, and your Product Manager: “Predict item availability”.

In the last 5 minutes, using Trello stickers, ask your team to vote for the best prediction problem. This is what your team will focus on for the remainder of the sprint.

*Note: It’s not uncommon that a technical team member may voice their opinion that this problem is not a suitable machine learning problem. If this is consensus, you’ve saved your company time and money. Not all problems are machine learning suitable.

2. Brainstorm the inputs/features of the model (10m)

Photo by Sam Chow

Goal: Develop a broad set of inputs for your model

Your team now has 10 minutes to brainstorm a list of all optimal inputs. Stress the fact that the team should not be limiting their thoughts with what’s possible/impossible. The goal here is to have a list of all the data that would be ideal for this model.

After about 6 minutes, begin asking the members of your team that are finished with brainstorming to organize the cards into themes. With the most important inputs placed at the top.

At the end of this section, you should be starring at a list of inputs that you’ll need to build your model. As an AI PM, this makes my life easier because now I’ll know what I’m looking for and will have to solve the next piece of the puzzle.. The “How”. Because this will be largely dependent on your organization, I won’t go into this here. I may write about this at another time.

Now you understand your prediction problem(s) and the inputs required to train a model.

Photo by Sam Chow

3. Brainstorm possible bias outcomes of each input feature (10m)

Photo by Sam Chow

Goal: To understand the biases that your inputs can introduce to your prediction

Starting with each input theme, start a new column and ask your team to brainstorm about the possible consequences of the inclusion or exclusion of this theme.

Answers should be provided as before, one thought per card. For example: a possible consequence of not including the input data “websites frequented” is that your recommender system may be incorrect, resulting in annoyed customers which leads to lower sales.

4. Summarize, Capture & Share (AI PM)

In 1 hour, you’ve just walked your team through a data brainstorm and now you have a tactical roadmap to move forward.

You’ve determined a number of prediction goals, the inputs you need, and most importantly, you’ve spent time thinking about the possible outcomes of of those inputs reducing the chances of unexpected predictions.

Use the remaining time to summarize the results with your team, capture the information and share.

I always like to issue a survey to the team to learn whether they enjoyed the process and learn how it can be improved. If you enjoyed this article and/or have suggestions, connect with me and let’s chat.