Why is the product delivery for AI powered products different? (Part 1)
AI is one of the most popular topics in Google right now. More and more companies are embracing or exploring AI to stay up to date or gain competitive advantage. The speed of the development in this area, interest and adoption of it might be the most rapid we have seen in a new technology.
Because we are learning on the fly and the landscape changes daily, there is still some improvement to be made in how we are delivering AI powered products.
In the first part of these series, I am going to focus on key differences between standard and AI powered products and identify key risks associated with each point. I personally faced or am facing each of those challenges and found out that overcoming them is not always easy.
In the second part of the series, I will look into ideas how can we address some of those most common pain points.
Difference 1— Data science team
First and foremost the biggest difference is that we have data science teams involved — yes that sounds obvious, I know, but I think the difference in ways of working and mindset of the team members is quite often neglected, that can lead to misunderstandings and delivery problems. The same is true for the engineering teams — the product teams need to find their own ways how to manage this crucial group of stakeholders and it should be definitely different to standard engineering team. They have their own methodology, their own way of thinking and the problem they are solving can be different.
Key risks:
- misunderstandings of the requirements,
- challenging stakeholders management,
- difficulty in incorporating data science ways of working into product delivery cycle.
Difference 2— Team set up/size
Another point worth calling out is team set up — suddenly we might have stand ups with larger teams who may not all be familiar with agile ways of working. The reason behind that is very obvious — we have more people involved, since we have other teams working on the product (Data Science and also MLOps). In this case, we need to be quite often more flexible, be more feature focused and try to organize ceremonies in a way they are useful but also not overwhelming. I’ve never been a person who thinks that framework should be followed by the book — frameworks are here to help set up the teams and make our work more efficient, but we shouldn’t let the framework complicate our lives.
Key risks:
- difficulty in setting up effective ways of working.
Difference 3— Research/discovery/analysis part of the delivery
This crucial part of data science work is, in my opinion, very different to a standard engineering spike. One of the biggest challenges is the fact that it is extremely hard to estimate the size of the work, timeline or story points — all the artefacts which are commonly used in product management, agile and delivery planning may not work here. What makes it even more difficult, is that there could be multiple ways of solving the problem and also the whole analysis might be unsuccessful and we need to start again. We should focus here on timeboxing and setting goals. We still can use Agile artefacts but we need to find a way how to use it consistently and efficiently.
To give an example - we would like to identify if using external data sources can improve quality of the model — we can define experiments which can help us determine the approach, but it is difficult to estimate how long each experiment will take without seeing the data and also we can realize that actually our human intuition was incorrect and all the experiments fail.
This part can be the most challenging for more junior product managers/product owners — it is important to be able to understand the results of the research and know how close we are to the goal — asking the right questions and following up with data science team are crucial here. In the example above, product team needs to be able to understand where to stop research and move on.
Key risks:
- prolonged discovery phase,
- unsuccessful results discovered very late in the process.
Difference 4— Maintaining the product in production
We all know that monitoring and alerting for all products is important, but the main difference is that models are “living” products — their quality can change over time, some results can be off or we might not be able to generate insights because of the change in data. This is all quite different to a standard bug in software in traditional sense. It is essential to not only have robust monitoring in place, but also have a process in place to deal with changes and maintain the quality.
Key risks:
- degradation of the production model what impacts clients,
- difficulty in incorporating monitoring processes in every day work.
Difference 5 — Clients education/usage of the tool
Selling the tool to the user is arguably the most important part of the product and software development process. Despite the fact that AI powered products are more and more available and a lot of people are interested in them, I still think that education on those products and explanation how they work is way more difficult than standard tech applications. These tools should be easy to use, but also Customers should be able to trust them and understand why the results they see look the certain way.
For example, education on how the forecast is generated and how frequently we can see the updates is crucial to set the expectations, especially when the big events happen which can affect future numbers.
Key risks:
- unsatisfactory usage of the tool,
- misunderstandings.
Above I have listed, in my opinion, 5 main differences why AI powered products are different from delivery perspective and can be more challenging for product managers/product owners. In the next part I will look into how we can mitigate some of the risks and overcome the challenges — stay tuned!
