AI Myths When Dealing With Newbies
As I interact with customers in my AI consultant’s role, there are some standard myths I encountered. Clarifying these myths appropriately is the first step in enabling the customer’s AI journey. Most of the problems arise from general AI illiteracy. Many companies, when starting their AI journey, ask their most experienced mangers (or the ones with bandwidth to spare) to handle the AI transformation. The trouble with this is — these managers are experts in traditional software development paradigm. Without a formal introduction to AI, they try to perceive an AI project from the perspective of classical software development. At a bird’s eye view, an SDLC approach looks very applicable to AI projects as well. But the similarity ends there, and devil is in details. AI projects have some significant differences which warrant a special process and understanding. To be fair, I myself have come from a classical software development background and have later moved into Data science via a formal education. I have suffered from some of the same wrong notions in the beginning days of my data science career. It is very necessary to disabuse your customers of these myths so that you can have smoother project experience.
Leaving aside the tactfulness and skill needed in conveying the knowledge, here is a list of the problems I have often encountered with such customers. The below list is neither complete nor comprehensive. But I would like to make a list of the myths I have commonly encountered.
Assuming data is sufficient to solve any problem
Myth: This happens when the managers are over reliant on the marketing hoopla regarding AI. They assume that once all the data that is being generated is dumped on the AI server, any problem can be solved by AI algorithms.
Reality: This is seldom the case. A classic example I would like to provide is from healthcare. Lets say we have the data regarding every hospital in the city regarding their number of monthly patients, number of staff leaves, vacations, number of syrenges used, number of healthy babies born, number of caesareans done in a month etc. This is great data. But this will not be sufficient to predict if a particular lady is going to have a normal delivery or a c-section. For achieving this, we need data at an individual patient level. If you dont have this level of data then no algorithm can yield the expected miracles of AI.
The approach I have taken is to ask the managers to answer this question. If an infinite amount of time is available, can the task be done using a pencil and paper using the available data? If the answer is yes, then the problem and data is eligible for AI solutioning. Complexity evaluation and ROI comes as a next step.
Assuming the AI models to be general purpose models build-once-use-forever
Myth: The model built for usecase A can be deployed for predicting results in usecase B.
Reality: It is not uncommon for mangers to throw terms like “evolutionary prototype” etc when dealing with AI projects. AI PoCs need not necessarily be “throw-away” prototypes. But it is entirely possible that a model built using a sub-set of data will not simply scale to a full data set (i.e a larger number of columns). If the data used for PoCs is not representative enough of the full data set, the PoC results can be misleading.
This myth is often encountered at the end of PoCs. An example I can give is — lets say you have a PoC to perform time-series analysis on a server’s log data. The code you have developed for data munging, performing Time Series Analysis and predict the KPI cannot be reused as-is for, say, stock market data. True, both of them use the same concepts like stationarity, seasonality, differencing etc. But the model built for analysing a server log cannot work for stock markets. Classic programming algorithms have concepts like fan-in, fan-out etc which can handle some of these complexities but not many AI solutions have matured to this level.
The above example might be an extreme and an obvious one. But a tricky one is — You are at the end of a successful PoC for analysing Windows server log. Manager walks up to you and says “feed Unix server log data to the same model and get me equally stellar results”. Truth is, there is no guarantee that the model scales as-is. The argument is “why cant we tweak the Unix data to have same column names and use the existing model?” It just does not work that way. As data changes, the model must change. The parameters which were very relevant for Windows server log might not be of any relevance to a Unix server. Simply changing the column names and data types will not give you the same results.
Using simulated data for PoCs
Myth: For showing a proof of concept of the product to a prospective customer, simulated data can be used.
Reality: This is the biggest problem I face and encoutering this command is my biggest fear. The reason is simulated data is generated using some variant of a random number generation. However, the random numbers generated in computers (can be any language) are not truly random. They are pseudo-random numbers. i.e. they follow a mathematical function. No matter how complicated the function might be, there is a pattern and predictability built into it. Now, machine learning algorithms are extremely good at understanding predictability. So, any simulated data can be understood and predicted with relative ease. It is not uncommon to get very high classification accuracy or regression r2. This provides a false view of the world and the PoC in question. Moreover, the solution used shall not scale to real world. Stochasticity is a defining charateristic of real world data. Noise is what makes a data real. ML is all about understanding this stochasticity. Predictability in the pseudo random numbers defeats the very purpose of doing machine learning.
The trouble is — explaining this to an AI illiterate is extremely tough. In the classical software development world, simulated data was almost always a trusted option for running PoCs. Asking a manager to not depend on his most trusted tool is a major change and those conversations never ended satisfactorily for me.
Mis-understanding r2, Precision/recall with SDLC’s testcase success/fail rate, Not understanding the trade-offs
Myth: We have a r2 (r-square) of 0.6. This means the ML model is successfull in only 60% cases and is wrong in 40% cases. Any code must meet 99% testcase accuracy. Else, it is useless.
Reality: OK, this one is a classic. This is second biggest of my fears when dealing with managers. In classical software development, coding is based on rules. These rules are formulated by human beings. Test cases are designed to verify the boundary conditions. Consequently, expecting a 99% success rate of testcases is very reasonable.
However, AI’s realm lies beyond manual rule definition. Crudely speaking, we throw the data at the AI engine and ask it to cough up the rules. And the stochasticity inherent in data makes it hard to achieve 99% success ratios. As of now, AI has not matured to levels of reaching 99% success rates whether it is regression or classification or anything else. AI, at the moment, is being used to reduce human effort. Not replace it entirely.
This is the source of all mischief. If the client is not familiar with this fundamental difference, then it is likely that they will equate your report of r2 or accuracy/precision/recall etc with that of traditional software’s testcases success ratio. I always found disabusing this notion to be the second most difficult activity. The result often for me had been that the client manager says “you don’t know anything. You should deliver six-sigma projects” etc. And you end up feeling like killing him or yourself.
Learning Python/R is sufficient to run AI project
Myth: The sentence is self-explanatory
Reality: Coding in Python/R/Julia/gobbledy-gook does not mean you are employing AI. Especially Python. I heard someone comparing Python to a Swiss-blade and found myself agreeing with it. Python can be used for diverse areas such as web development to GUI to android app to HMI. While Python/R/Julia have excellent support for AI algorithms, AI body of knowledge has nothing to do with these languages. AI can be implemented in purely C, C++ or even FORTRAN. Saying that using Python is implementing coding is akin to claiming that using Java is necessary and sufficient to implement multi-threaded or multi-tasking programs.
Unable to understand version control issues
Myth: We only need to version control code in AI project.
Reality: Not so. Version control in AI requires lot more than that. We have to version control the datasets, code, training and testing subsets, feature engineering changes both individually and also together.
That said, this is a relatively easy myth to bust. No manager disagrees when we say we need more-than-usual version control for AI projects. A careful explanation of the reason should normally suffice. Many times, it makes the managers feel better to have learnt some specialities of AI projects in easy terms. So, this should be a generally easy myth to bust.
Not focusing on explainability requirements of AI solution
Myth: I have a great AI solution with wonderful r2 or precision/recall/accuracy metrics. I am all set. That’s all I need.
Reality: You also have to consider explainability. Simply saying that “AI has yielded this solution with great accuracy” will not cut it. For example, lets say you are approving or rejecting a loan application using AI. If you can’t explain why the algorithm has rejected a particular request then, you cannot really deploy it for real world where legal implications are a major concern. Explaining the un-explainability of AI algorithms, especially with deep learning algorithms, is an aspect which needs consideration upfront.
This list is by no means exhaustive. Its not even authoritative. These are some of the recurring myths I have encountered as an AI consultant. What else did you encounter?
