The AI Hierarchy of Needs
While learning about product management for Data Science and AI, I came across an interesting concept… The AI hierarchy of needs.
It is a guide for companies wanting to build intelligent systems amidst all the current hype. Below is a very simplified representation of the AI hierarchy of needs.
Many companies want to build AI, but in reality, they aren’t yet ready for it. Several factors have to be met before good, ‘ethical AI’ can be introduced into any system. The AI hierarchy of needs follows the same principles as Maslow’s hierarchy of needs in that, the conditions at the bottom of the pyramid have to be met before those that are higher up.
Data is the currency of AI, and for AI to be built, we must collect data. Without data there is no AI, thus, it is clear why the base of the pyramid is ‘DATA COLLECTION’. Data can come from anywhere; sensors, images, logging e.t.c. Without a large corpus of representative data, companies usually won’t be able to build AI, hence companies must take time gathering data.
Once we have our data, what next? The data has to be stored somewhere, in a form that is consistent, relatively easy to understand and readily available for use. This is collectively called ‘DATA INFRASTRUCTURE’ inclusive of data pipelines, ETL, and storage. Data engineers are particularly important at this stage to ensure the data is ready for the higher levels of the pyramid.
At this point, it is time to put the data to use, but first, it has to be cleaned and prepared. Anomalies should be detected if any, and adjustments should be made to our data collection and infrastructure methodologies. The data should be analysed and insights gleaned. At the end of this stage, if AI is our goal, we should have training data ready on which our machines can learn.
Following data analysis, generating features and labels e.t.c we can introduce data science. At this level, we can build models using simple algorithms e.g a regression model, and test. It is important to note that ‘simple’ here doesn’t mean poor or bad and it is not unusual that these ‘simple’ models end up being so difficult to outdo. This stage also allows us to do a lot of experimentation, feature engineering and/or creation and the metrics got here usually serves as a baseline when we want to build more complex models. Enter…. AI.
We are finally at the top of the pyramid. We have perfected our data collection strategy and have gotten a hang of our infrastructure. Our analysts and scientist are cleaning up the data and gaining insights. We have an understanding of what features give us the best improvements and we have a baseline of what we expect from our more complex deep learning models.
As a PM at a company wanting to build AI, it is important to know your company’s current position on the pyramid and ensure standards are set and kept across every level. It is also important to note that building AI is a risky business and at the same time, time-consuming. You might spend years building an intelligent system only for it to fail miserably. Ensure you communicate this to your stakeholders.
Lastly, it is important to build ‘ethical AI’. Intelligent systems should never perpetuate inequality. There are several notable examples of AI perpetuating bias e.g Amazon’s hiring algorithm was biased towards hiring men when it comes to technical roles, this was because most of the data it was trained on had males in technical roles amongst other reasons. This article contains more examples of this.
Finally🙂 , whenever you're building intelligent systems, remember the hierarchy of needs, make sure you have a solid foundation and never forget about ethics.
Go ye into the world and build kickass AI. 😎
If you want to learn about product management for AI & Data Science, click here. As usual, I wasn’t paid for this. I took the course on udemy and I loved it. Cheers.
