What Exact Gap Did AI Fill? Can You Explain Machine Learning In Simple Words?
Who is this article for?
This article is for someone who wants a non-technical way to understand why/when AI is needed in the first place and how Machine Learning (which is a type of AI) solved for the needs. Machine Learning is one of the foundations of AI. The next level of AI is Deep Learning and even more advanced level of AI is LLM or Large Language Model. Those advanced topics will be explained in separate blogs.
Letâs start with the âWhyâ â Why and When exactly is AI needed?
Artificial Intelligence, like any technology, is meant to automate tasks to save time and money and add comforts and conveniences. Broadly speaking, there are two types of tasks â tasks that are well-defined and those that are not. The âwell definedâ tasks can be automated with a software program but when it comes to tasks that are not âwell-definedâ, software programming falls short and thatâs where we need âArtificial Intelligence. To build that âintelligenceâ, there are multiple techniques and one of them is Machine Learning. We will get to that technique but first letâs understand what makes a task âwell-definedâ vs. one that isnât. To understand, letâs use the example of a bank from the 70s when all the banking processes were still fully manual.
Processes which are âwell-definedâ
(Jump to next section if you know how traditional programming can solve for well-defined tasks)
A well-defined process means one which has a clear set of instructions. There is no involvement of any âguess workâ or âestimation, prediction, forecastingâ or âany visual inspectionâ or any âgut checkâ in the set of instructions. There are clear instructions which are used to process a given set of provided inputs and a clear set of outcomes (outputs). Examples are some tasks that a bank teller used to perform in the 1970s. For example, if a customer walked in, then as âinputsâ, they provided the following â the type of request (say cash withdrawal), details like account number/amount, and an ID for verification. The teller was âinstructedâ to do the following â verify the ID, if verified, then check their bank paper files to see if there was an account with that number and if yes, check if it had enough balance and if yes, then disburse the cash and reduce the bank balance by that amount on some paper form. Similar âinstructionsâ were defined for depositing checks or providing account balances. Barring the âID verificationâ which required a visual inspection and hence a âgut checkâ, each of the other steps were clear with no ambiguity. They could be written down in a âif X, then do Yâ type of instructions. Software programming is basically about automating these kinds of tasks. Since the 70s, this entire Information Technology revolution has largely been about automating tasks like these in every industry thatâs out there. For example, retail shopping got automated because you just provide an item name, quantity, your address and payment information as input then the software finds a matching price with discounts, adds tax, charges the card and instructs the warehouse to ship the item. Each of those steps is split down into a smaller and smaller but very fixed and clear set of instructions to make all of that happen. Such examples are the âwell definedâ tasks which can be automated with traditional software programming, without the help of any AI.
Processes which are NOT âwell-definedâ
Banks do a lot more than these âwell definedâ tasks. For example, bankers approve loans for homes. Bankers also provide investment counseling â where to invest, what the risk is and so on. Bankers also process an ad hoc set of requests through their âcustomer serviceâ counter. Each of these tasks has several elements of âgut check or judgment or estimation or predictionâ involved â the aspects that traditional software programs cannot handle.
Letâs start with the first scenario of Loan Application. Let's say a customer requested a loan and provided all his information like salary, household income, net worth, number of dependents, other open loans etc. Now the loan officer has to look at all these factors and estimate the probability of this customer defaulting on the payment. If the probability is not extremely low (it can never be 0%), then loan will be approved. Back in the 70s, the loan officer had to âguess or estimateâ that probability based on his gut or judgment. Now in the 80s/90s, after automating the well-defined tasks of the Bank Teller, the software engineers asked the Loan Officer â âCan you describe steps you follow so I can write a program to automate those steps for youâ. The Loan Officer replied saying â âI look at all the information in the Loan application and then use my judgment to estimate how likely the customer is to defaultâ. The software engineer said â âWell sir, thereâs no way to put your judgment in a software programâ. The result was that the Loan Application processing remained completely manual, unlike the teller tasks like âcash withdrawalâ or âaccount balance inquiriesâ.
Same thing happened when the software companies tried to automate the job of the investment counselor in the bank. The investment counselor said âI look at stock market, the company financials and some charts and then the risk profile of the customer, their retirement age and their investment goals and then use my judgment to recommend an investment planâ. The software engineer asked â âwhy donât you follow a set of rules to recommend investmentsâ and the Counselor said âI canât because thereâs way too many factors. Factors about the customer, the companies to invest in, where the stock market level is, the current economy, the currency exchange rate, the political news and so much more. How do I make a rule around all of that? Even if I do, it's still all a guess. My best guessâ.
The situation was even worse with the customer service (CS) desk. When the software engineers told the CS guy to describe the rules or instructions he follows to do his job, he said, âI just use common sense. People come here with all sort of things. One man came here panicked because of a charge on his account statement. Turned out it was by his son. One woman came and talked about her wedding for a while and I had to just guess that she wanted her maiden name changed on the account. Another person wanted to block his estranged wife from having access to their joint account. These are all so random scenarios. How can I give you a clear set of instructions. You just have to use common sense.â Clearly, this Customer Service process couldnât be automated either.
In a nutshell, traditional programming falls short when there are no clear set of instructions or when there is prediction or estimation or judgment required. This typically happens when there are way too many factors involved in a decision and when there are very complicated relationships between them which can be put down in a simple âif X, then do Yâ type of steps.
Thatâs Where AI Steps In
The Software Engineers left the bank saying â âwe automated all your well-defined tasks but for the rest â you guys keep your people because we canât replicate human judgment in a programâ. This is when the AI guys stepped in to say â âlets give this a second shotâ. AI guys means Data Scientists. They reached out to the Loan Officer and said, âHey, you say that this is all your judgment, but can you explain how you built that judgment? When you started your career, you didnât have this judgment, rightâ.
The Loan Officer said â âOf course not, I didnât know anything at that time. It's been 35 years for me at this bank and I have built this judgment over these 35 years. As I started approving loan applications, I would follow their mortgage payments on the subsequent years and especially started looking at all the people who defaulted and couldnât make the loan payments. I started looking at what was common between all these defaulters and observed a few patterns. For example, a lot of defaulters were the ones who had defaulted on their credit card payments or other bills. But when it came to marital status, singles were likely to default more but just a little bit more than married couples. This told me that some factors carry more importance than others in predicting the default possibility. What I mean is that the marital status didnât matter much but bill payment history said a lot. This way Iâd keep refining my thinking about a variety of factors and in my head, Iâd form a judgment about what was important and what wasnât. Thatâs it â Iâd then have these judgments at the back of my mind while processing the applications and Iâm known for being fairly accurate in my predictions.â
This is where the Data Scientists said â
âWell, this sounds more like math than some magic which only humans can do. We can make a computer do this!â And what they came up with as a solution is now called âMachine Learningâ. So, what is it exactly?
To the Loan officer, the Data Scientists said â âGive me a list of all the customers who took a loan from the bank along with all the details collected during their loan application (like their salary, their history of missing payments on bills, their salary, the list of all their loans, their family details etc.) and also tell me which of these customers landed up defaulting on the loanâ
When they got this list, they wrote a computer program to calculate how frequently a âhistory of missing payments on billsâ was already on the list of defaulters at the time of their loan application. That turned out to be the case in 80% of the defaulters. Clearly, if this factor had been given enough weight, a lot of defaults could have been avoided. Then the program found that âsingleâ marital status showed up in only 20% of the defaulters, which indicated that marital status was not really a helpful indicator. They did this for all possible factors collected on the application and now it was clear which factors mattered more and which mattered less. In other words, the computer program now had the same insight that the Loan officer was using to make his decisions. To âstoreâ this learning, the Data Scientists asked the computer to assign different levels of importance to each factor based on frequency at which it appeared on the list of defaulters. This level of importance was called the âweightâ of that factor and it was basically a number between 1 to 10 where 10 is a very strong indicator of default and 1 is very low indicator of default. For example, when a person had a history a late or missed payments, the computer would add a score of 8 (out of 10) to the customerâs âdefault probability scoreâ and if someone was single, then it would add just 2 to their score. When all such âweightsâ were added up, the loan applicant would get that final default probability score. The higher the number, the higher the chances of default. This computer program which memorized all these âweightsâ against each of the factors (also called attributes) was called the âMachine Learning Modelâ. And because Machine Learning is a basic form of AI, its sometimes also called the âAI Modelâ.
So, the eventual solution for the loan application process looked something like this: When a customer filed a loan application, all the details (like salary, history of missed payments etc.) were the âinputsâ to the ML model. The ML model would apply the right âweightâ to each of the input factors which would generate the âdefault probability scoreâ. One aspect worth noticing about this ML based solution is the presence of this additional âweight layerâ between the input and the output layers. In other words, presence of just two layers (input & output) is what traditional programming is about while ML adds that third layer of weights.
Anyway, so in our example, the Data Scientists went to the Branch Manager to say that their ML model can now replicate the judgment of the Loan Officer. In fact, the ML Model is more accurate because it can remember and calculate these complicated details more accurately. In this simple example, this approach of âMachine Learningâ was predicting the probability of loan default with the same level of accuracy as the Loan Officer and perhaps even better than some other Loan Officer who are less diligent than others.
Summary and Conclusion
In summary, traditional programming can solve tasks which are well defined. You give a set of input, and the computer will give you output based on a set of ârules or instructionsâ which look like âIf X, then Do Yâ.
When the task is not well defined or when you need judgment or ability to predict or estimate â traditional programming cannot handle it. To be able to do handle those scenarios, a computer first needs to analyze data and learn how much each of the input matters and to what extent. This âextentâ is stored as a âweightâ of that particular factor and the collection of such weights is the ML model. The ML model applies these weights to each of the inputs to make the prediction or the estimate. This approach of learning from data to predict or estimate is called as Machine Learning.
Lastly, while this ML approach can handle some business scenarios like the Loan application scenario, it falls short of giving adequate level of accuracy to handle the âinvestment counselingâ process of the bank mentioned earlier and is even less accurate when applied to the customer service process in the bank. For those scenarios, we need more advanced form of learning to build the right AI model and that technique is called Deep Learning. A similar article on Deep Learning to follow soon.
If you found this article helpful or interesting, please Clap or Follow!