Your Guide to Executing AI and Machine Learning Projects

There are a lot of complexities when it comes to building, shipping and executing AI and machine learning (ML) projects. In our previous article in this two-part series, we looked at the importance of collaboration and everyone understanding their roles within the product management team — and how these are the key ingredients that create the conditions for success with an AI project.

So why does collaboration between product management, data science and engineering really matter?

Product managers need to ensure that data scientists are delivering results in ways business counterparts can understand, interpret, and use to learn from. This includes everything from the definition of the problem, the coverage and quality of the data set and its analysis, to the presentation of results and follow-up.

Only 13% of data science projects, or just one in 10, actually make it to production. VentureBeat

While keeping the team focused, product managers collaborate and brainstorm together with data science and engineering to clarify what to do with the information collected. The nature of ML and AI forces teams to work collectively since the risks are far greater. It gets them to communicate better and a common, shared understanding of the end goal develops organically.

It’s a bottom-up process, you attempt to solve a problem with the team, get a signal early and use it to construct the bigger picture. By knowing your training dataset cold, you set the vision and direction of the product.

Being able to construct summaries and visualizations that examine why the model works illustrate not only the importance of validation to improve a product but that teams always prefer being led by someone who invests in a baseline to understand the data.

Building and Shipping AI and ML Projects

With the concept of effective collaboration in mind, what more do you need to ship AI-enabled experiences? Unfortunately, most product teams aren’t well equipped to handle the end-to-end delivery of an AI/ML project. These require complex rethinking and changing product and workflows in fundamental ways. Don’t expect it to behave like a traditional software project.

AI/ML project generally have longer timelines, more costs and contain a number of moving parts that can sometimes be challenging. Here are some of the costs to take into consideration:

• Engineering costs: This includes new infrastructures, tools, processes and GPU costs.
• Data costs: This is for data collection, data storage, data pipelines, data preparation and cleaning (with the right features and labels).
• Training costs: This includes the training model, needing to frequently revisit the post-launch data and ensure the ML model remains up-to-date, training run time, monitoring technical glitches and prediction accuracy.
• UX costs: This is for testing to see if users respond positively to machine learning (i.e. in-product feedback) as well as mitigating wrong prediction catastrophes through business logic and UI safety nets.
• Maintenance costs: This is for the continual monitoring of AI performance, development of new training set data, oversight mechanisms, maintenance procedures to update the models — this will help adapt to unforeseen changes in its environment.

Furthermore, it’s important to understand that there’s a cost to getting it wrong with AI and machine learning. It is therefore up to the product managers to anticipate and recognize the consequences of not delivering value on the organization’s investments in AI capabilities.

From Analytics to AI

We’re closer than ever to creating some of the most innovative and smart AI technologies the world has ever seen. Below is a roundup of advice on how to understand the abstraction of data, and how this can help guide you through the process of executing your own AI project.

• Get the right data set: Look at your problem space and collect the unique data source first. Label data and generate a stronger dataset. Then constantly look for additional sources of data that will continue to improve your product.
• Opportunity assessment: You’ll need to first understand where the biggest improvements can be made. The most important element of data gathering is what to do with the information once you collect it.
• Focus on eliminating bias: Besides collecting the right data, another big step is ensuring the data you work with is correct. Bias is potentially present in any dataset and it’s up to you to be aware of potential bias and work with a data scientist to resolve it.
• Weigh the cost of getting it wrong: Cover all scenarios that the Machine Learning Module (ML) might have to encounter. It’s important to understand what errors look like and how they might affect the user’s experience of the product.
• Know how you’ll know if you are successful: Have a solid problem understanding. Link and correlate outputs back to the input data. Outputs should correspond to your intuition. Note how many times the ML gets it wrong in comparison to a human.
• Build a safety net as well as a feedback loop: Safety nets are visible to users (UX) and take into account that erroneous actions can have consequences. Feedback loops allow you to monitor prediction accuracy, etc.

When what to do with the information collected becomes clear, engineers can figure out what tools and models are necessary to solve it. Only then will they iterate and fine-tune models to deliver accuracy. They will seek optimal performance because the product manager created model interpretability — the ability to verify what the model is doing is in line with what they expect.

Looking to get started on your next AI project and need some guidance, coaching or mentoring on product management? Check out a number of our workshops and other offers or contact us, and we’ll see how we can help!

Missed the first article in this two-part series? Catch up here: ‘Tomorrow’s Product Managers Need to Create the Conditions for Success With AI’.

Bain Public offers consistent roadmap planning processes & tools for business leaders and product managers organized around what motivates, inspires and improves growth. We offer a variety of blog posts, e-books and approaches designed to help organizations understand their digital strategies, introduce elements of roadmapping and establish product-led change amongst the senior leaders and managers. Our approach, product, expert advice and coaching help entangle complex technology, people and roadmap dynamics. For more information, visit bainpublic.com or call 514–442–8487.

Thanks to Paul Ortchanian for contributing to this article as well as reading drafts and overseeing aspects of its publication. Also, if you have any feedback or criticism about this article, then shoot us an email at info@bainpublic.com.