A Product Manager’s Guide To Build An Enterprise AI Product

During his “Nuts and Bolts of Applying Deep Learning” Deep Learning school talk in 2016, Andrew Ng made an interesting comment about product managers building products based on Artificial Intelligence.

He said, product managers or business people often ask him, “What can AI/Deep Learning do for their products?”

And some product mangers he knew of assumed that AI can pretty much do anything. In the talk, Andrew then went on to set expectation of AI capability for product managers. His simple principles were:

First: If a typical person can do a task in less than one second, AI could be used to automate that task

Second: AI could be successfully used to predict outcome of the next nth sequence of events.

We are in 2018 now and although majority of major success stories in AI coming from the largest tech leaders of the industry (Google, Facebook, Microsoft, Baidu, Apple, Amazon), adoption of AI in enterprises during 2017 remained low.

There is a tremendous interest in embedding AI into enterprise product offerings. Mega software vendors like SAP, Oracle, Salesforce are leading the pack and many SaaS providers such as Adobe Systems, Tableau, Splunk, Workday, Zendesk are closely following.

Although “What AI/Deep Learning can do” question is somewhat well understood, Andrew’s comments about lack of established workflows for product managers building AI applications for enterprises still holds true.

Well-defined product design workflow (product managers discovering customers, building wireframes, designers building visualization and developers implementing design) that worked fairly well for desktop and mobile era, would need new enhancement.

There is a famous saying in product design community — “You can’t innovate on products without first innovating the way you build them”.

Here are some lessons learned from my personal experience of building enterprise ML/AI product in startups and enterprises.

Product Management Craft — What Stays Valuable And Applicable

Product management core competencies and emotional intelligence (EQ) which make the best PMs create products with strong user adoption and help generate exponential revenue growth are still the primary factors for AI product management success. Conducting customer interviews and user testing, performing market assessments, translating business-to-technical requirements, defining and tracking success metrics, pricing and revenue modeling — principles as sketched in this HBR article are the essential ingredients of core competency.

Identifying and articulating the business value AI adding to the product is most critical of all. Value to your own business is where AI has been applied extremely successfully by tech giants. Reducing customer churns, increasing customer engagements, predicting sales deal closure, optimizing inventories and supply chains — these are the common values AI has been adding to one’s own business. Many AI applications targeted to enterprises are now offering these values to customers.

The AI killer app for enterprises today is cost savings. Customer service, back-office tasks such as human resources, finance and accounting are perfect candidates for the tasks that would be automated with AI. If you are a product manager building such applications for enterprises, cost savings and improving customer experience are relatively easier metrics to quantify. For example — Salesforce Einstein helps enterprise customers use their own historical lead and account data to predict which deals are most likely to close. A very clear business value AI is bringing to Salesforce customers.

But, if you are a product manager building AI/ML automation products that eat away your company’s professional services or managed services revenue, be certain to understand the value transfer when you are building your pricing model.

What’s New And What Needs Innovation In Product Management Process

Here is a list of few items, mostly inbound (engineering-facing) activities that I feel are different about building an AI-product for enterprise.

Enable a strong yin-yang relationship between domain expert and data scientist : In some startups, domain experts drive data scientists and data scientists are mainly responsible for prototyping and testing. Prototyping and testing involves deciding data training set/validation set/test set, feature extraction, testing multiple algorithms and iterating on the results.

Best outcome happens when there is a joint discovery and shared accountability between domain experts and data science team. And it’s usually product manager’s responsibility to bring them together to solve a business problem.

The model-building process is quite different in enterprise products as compared to consumer applications where data scientists can experiment and iterate quite independently on training algorithms to detect faces in videos or identify hand-written alphabets in postal codes or identify objects from Pinterest images. For enterprise AI products optimizing process or finding outliers need extensive help from subject matter experts. As an example, IBM Watson Knowledge Studio is built to enable enterprise domain experts to teach Watson to understand nuances in unstructured text.

So, as you are building your enterprise AI product, make sure subject matter experts and data scientists are collaborating well.

Apply business judgement on choice of machine learning approaches based on performance vs. data volume : Not all enterprise products need to train a large, multi-layer neural network to obtain the best performance. Actually, for a small a set of training data, traditional machine learning models work as good as a small/medium sized neural network. When the available data is large enough (mobile, IoT, internet, satellite images), large neural nets have the capacity to absorb that data and perform in superior manner.

Understanding where traditional algorithms do a good enough job vs. where deep learning becomes a must-have for performance boost, is a product management responsibility.

In trading or banking applications, incremental gains have enormous business impact.

A ML/AI model has many tunable hyper-parameters that affect performance. This could be as simple as number of trees in a random forest or the kernel in support vector machine or could be as complex as learning rate in gradient boosted or deep learning methods. As an example, a four-layer neural net to perform a 2D binary classification could have 20+ tunable hyper-parameters including traditional ones such as learning rate, activation function as well as regularization, architectural parameters and feature transformation parameters. Finding the best parameters that have significant performance improvement for the end result is a time-consuming and iterative process and involves grid, random, local or expert intensive manual search.

If optimization improves your product’s business metrics, do your research on outsourcing fine-tuning of model parameters as well as leveraging as much as internal data science resources.

Understand the accepted accuracy of AI/ML algorithms in your product : Higher Precision vs. higher Recall or less false positive vs. less false negative — which one you would prefer depends on the types of business problems you are solving.

Let’s say, you are processing incoming emails to an enterprise and classifying them as suspicious vs. authentic, aiming for higher recall but lower precision would generate more false positives and delay the process of authentic email delivery but would make sure no harmful emails get through.

On the other hand, if your product is building predictive maintenance of a system or device, you would want to make sure you have captured a high-confidence failure scenario before creating lots of system alarms (which usually gets ignored) — hence opting for higher precision.

There is always a tradeoff and it is certainly a product/business decision.

Go easy on scrum process— you will kill creativity : Ok, scrum process is a much-debated topic in general and the success of the process usually depends on multiple factors including company culture, team size and past history of product development. Scrum methodology for software development have been very successful bringing in feature velocity, rapid prototyping and ability to pivot relatively painlessly.

But AI/ML are time-consuming processes and there is a fair amount of discovery involved in the process. Tying your data scientists to a rigorous scrum would not derive a favorable outcome for your product.

There is a fine balance between meeting your product milestones vs. delivering a superior product and most good product managers are quite capable of dealing with the tradeoffs. I have seen AI projects going from a series of failures to an overnight success with relentless experimentation and iteration by data scientists.

Particularly, for AI products where data synthesis is a major obstacle to get the algorithm to work, being patient on experimentation of approaches to create synthetic data is absolutely necessary. An example would be months of experiments conducted by Stanford students to create high-quality synthetic data from house numbers in Google street view images (Street View House Number — SVHN dataset) which improved the craft of end-to-end deep learning for reading text from photographs significantly.

AI On AI — Paradigm Shift To Accelerate Product Manager Productivity?

Finally, I want to leave you with two examples on how AI is being used to increase productivity of the other ‘chefs in AI product kitchen’ — namely data scientists and UX designers. This should fire up our imagination on how AI can be used to enhance product manger’s productivity.

Airbnb engineers came up with automated machine learning (AML) that automate the repetitive task of data scientists — exploratory data analysis, feature transformations, algorithm selection, hyper-parameter tuning and model diagnostics.

Airbnb is also developing a new AI system that will empower UX designers to literally take ideas from drawing board and turn into actual products. Once a design house feed their carefully defined components and guidelines into AI, it could be used to classify them and turn them into actual code.

It might not be long while AI could turn field notes from product team or customer conversation from account team into meaningful product insights. In the meantime, aspiring AI product mangers — brace yourself to build a good testing framework and iterate.

In my next blog, I will talk about some of the Go To Market challenges of an AI product — what will make users to embrace or hesitate to use an enterprise AI product?