Product Managers in AI products
Do the latest technological techniques rival the role of Product Manager?
Times change, technology advances, and today, we are more immersed in the Information Age than yesterday.
Information in huge quantities! As specialists in Digital Products, we cannot take it lightly. Increasingly, this will be essential for companies that want to survive in a competitive environment. As Product Managers, we need to make sure that we take advantage of existing internal data or collect external data to build better products.
Let’s see next these roles in action!
We are certainly used to visualizing the triad of dimensions to which we, as Product Managers, must pay attention: Business, Technology, and UX; well, the Product Manager of AI products must deal with a fourth additional dimension: Data.
Therefore, a Product Manager of an AI product must now become familiar with a whole set of new technical concepts, some related to the statistics field. In addition, I will now begin to relate to Data specialist roles: Data Architect, Data Engineer, Data Scientist,…
Notice that I say become familiar and not necessarily become an expert.
Normally, the explanation of the Product Manager role includes some similarities such as that of Orchestra Director, a profile that, without necessarily being an expert in the different fields, must be able to communicate with everyone and make them work as a single unit. This is still the case with AI Products.
Responsibilities of the AI Product Manager
Let’s briefly check some of the AI Product Manager responsibilities.
Define the problem to be solved.
Stakeholders may want to jump on the AI bandwagon for fashion in the same way that they wanted to jump on the metaverse bandwagon, before the agilism (the practice of Agile) bandwagon, and before that of digital transformation… Although it is very legitimate to want to explore all the possibilities of growth of a company, this must be done by applying common sense.
It is possible that the use of AI is NOT required to solve an identified problem. It is the responsibility of the Product Manager to fully understand all the edges of a problem and evaluate together with his team the solution alternatives.
It could even be that we, as Product Managers, have to change the way we approach the matter. This is because more and more companies, in an effort to stand out from the competition and innovate, try to take advantage of the data that they have been accumulating for years.
For this reason, it is not uncommon that instead of starting from a specific problem to which one tries to find a solution, one starts from a set of available data for which one tries to find the best possible use.
Industry and context analysis
It falls within the scope of the Product Specialist to adequately understand the context of the industry and the specific market. We must analyze the consequences of not implementing the solution or of not doing it on time.
We can use tools such as the SWOT matrix or a study of the competition to find out if they are already using artificial intelligence.
Data availability, Variable understanding and AI system selection
AI is not magic but science, and a system is only as good as the quality of the data used to train it.
As Product Managers, one of our main tasks continues to be applying common sense, extracting the necessary information from the experts in the field to be treated and, if necessary, identifying the variables to take into account to help the system focus its efforts.
Depending on the type of problem to solve and the data, time and budget restrictions, we must opt for one AI system or another.
What data do we have? Are they enough? Can we get more? At what cost? Are there open data sources available to anyone? What legal restrictions should we know?
We must take into account all the possible sources of data and determine if they are available if they are adequate, their cost, and their exclusivity…
(NOTE) It is said that many AI models have been feeding on all the information available as Open Data on Twitter through scrapping; as a result of this, they have decided to impose the current restrictions and limits on accessing their content.
Are they suitable for our target audience?
If our target audience is customers from Spain and we train our AI model with data from the US market, would it offer good results? As in any other product, it is essential to understand well what type of users our product will be aimed at and carefully select the data that we will use in our model.
Do we have the necessary technical capacities to be able to collect them, clean them, transform them, store them, analyze them…?
We must assess the maturity of the company in terms of data. Do they have the infrastructure and sufficient personnel? Perhaps it is not yet the time to consider the internal construction of an AI product, and alternatives offered by third parties should be evaluated.
Expectation management
The development of an AI product requires a lot of research and experimentation. We need to train the system with different data sets and statistical methods in order to find the best combination that will make viable our AI model.
So it becomes very difficult to accurately establish a delivery schedule, something that we will have to explain internally.
That is why sometimes the framework selected for development is KANBAN and not SCRUM.
A point that is usually the subject of discussion is the one related to the performance of the model. Stakeholders will tend to place high expectations regarding the expected results. It is our responsibility as Product Managers to set the minimum performance for an AI model to be viable, and this will depend on each particular case. The degree of human interaction in the process is also something to take into account: an AI product designed to improve human decision-making or an absolutely autonomous system will not be the same.
It is the responsibility of the Product Manager to provide these explanations to the stakeholders.
The strategy
As Product Managers, we must determine the steps to follow and make some decisions:
- What minimum data set do we need to train our model? that is, what will our MVD (Minimum Viable Data) be?
- Do we need to develop some functionality in our product first to collect the necessary information to later develop our AI product?
- How are we going to test our model in a production environment?
Metrics
In addition, as Product Managers, we must work on selecting the metrics that we will use to evaluate our model. We must take into account tools such as the Confusion Matrix, which takes into account both the successes of the model and the failures.
This specific tool allows us to identify the performance metrics of our model to which we must pay attention based on a mapping of the user experience according to the results obtained. Compares the actual and predicted values, assuming there are only 2 possible values: Positive or Negative.
Let’s see an EXAMPLE to understand it better:
Let’s imagine that we have been hired by the National Association of Mycology in association with the Ministry of Consumption to develop a mobile application that uses AI to determine whether or not a mushroom is edible. The objective is to provide a tool for mushroom pickers to reduce the number of intoxication cases per year and, at the same time, encourage healthy habits with the consumption of this type of food.
We design a system through which the user uploads a photo of the item to be evaluated, and the application labels it as Edible or Not Edible.
- Accuracy: Measures the hits generally, regardless of one direction or another. (TP+TN)/(TP+FP+TN+FN)
- Precision: How many Edibles there are among those identified as Edibles TP/(TP+FP)
- Recall: How many Edibles have we identified of all the real Edibles? TP/(TP+FN)
- Specificity: How many Non-Edibles have we identified of all the Non-Edibles TN/(TN+FP)
For this particular product, it is decided that the objectives of the AI model should be to minimize False Positives while keeping False Negatives to a minimum. Therefore, the fundamental metric will be Precision and, in the background, Sensitivity.
The Product Manager and his team meet with various mycologist to try to find out what variables are taken into account when classifying a mushroom. Various fundamental morphological characteristics are identified: shape and size of the cap, ring, gills, volva, stipe…
Fortunately, we have a large repository of images to train our system. Trying different combinations, we obtain 3 different models:
So, which model is the most suitable? Is the performance adequate?
Model 1 is discarded because despite having a high Precision, the Sensitivity rate is too low.
Model 2 has an acceptable Sensitivity, although its low Precision makes it unacceptable. Model 3, even though it has less Sensitivity, has much better Precision.
However, none of the 3 models is considered viable yet, as the Precision rate is still considered too high and a health risk.
By reconsidering the model variables, others not initially included are identified (habitat, season of the year, mushroom smell) and that can help improve the performance of our model. This is information that cannot be extracted from a photo, so a form is attached to the product so that the user can include this information. When we include it in the model, these results are obtained.
Although work must continue to improve the results, it is considered that the model is already viable, and the first version of it can already go into production.
Taking everything into account, we can assure you that the role of the Product Manager is still a determining role to carry out the development of an AI / Data product, from the conception of the product, through its development, deployment, and subsequent follow-up. Fortunately for us as Product Managers, there are still specific questions to be answered, decisions to be made, and opportunities to apply common sense when necessary.
