Julio plays sing-along — unraveling cross-validation.
Cross-validation is a core concept of Machine Learning and is used in each and every modeling exercise. The roots of this widely practiced exercise can be found in basic logic, Julio realized it during a game of sing-along.
It was a warm Friday night, and Julio was with all his friends. With some chilled beer in hand and pizza box on the center table, all the friends were reminiscing about the ‘good old days.’ They were talking about how cheap things used to be; how simple life was before the onset of the hyper-communication era; how great the old movies and music were. It was not long before all the friends were singing rhapsody of nostalgia.
As a group of music lovers, they started playing a musical game. The rules were simple: one member will play a song on his phone, and others will have to recognize the singer. One who recognizes the most singers wins. Julio was one of the oldest members in the group and hence agreed to mediate. He was a huge music lover too and had a rich knowledge of old and new songs. It made him an ideal person to conduct the game.
He realized there can be two strategies: One, to remember all the songs and their singers. The plan is nice for someone with a great memory, but the amount of time needed to memorize all the singer-song list is a lot. There is probably more cost than benefit to achieving this. Almost all the songs will have to be heard, and even after that, any new song might just come up in the game.
The second strategy could be to learn to recognize the voice of a singer. This might work. This strategy not just enables them to identify familiar songs, but also those not heard before. But the problem is to ‘learn.’ It might be more challenging to learn to identify the singer from the voice than to remember. Julio thought, “if anyone can remember the voice of their family and friends, why cannot they remember the voice of a few singers as well. It’s totally possible!”
Julio wanted to make it a little more fun for himself and wanted to conduct a cognitive study. Besides trying to run the game fairly, he also wanted to recognize which friend is using which of the two strategies.
Julio opened a music app on his smartphone and created a list of songs. He kept many songs on the list, some common and some lesser-known; some happy and some sad; some old and some new. He wanted to equip himself with a wide range of songs, he was hoping to analyze the recognition pattern to get some answer for his cognition project.
The game begins.
Julio was playing old and familiar songs in the starting; everyone has heard them, and so all were able to recognize the singer to all of them. At this point, Julio decided to shift the gear, he started playing the lesser-known songs, almost all of them released decades ago. Even at this level of difficulty, most singers were instantaneously recognized. Julio was quite impressed with his friend’s skills. It looked like as if they have been training for this night all their lives. But this also was bad news for his cognitive study. He knows no difference between those using strategy one from those using the second.
He decided to up the ante, he bought in the new songs. These were the songs about which he knows for sure that they are mostly unheard of in this group. A grin appeared on his face as he realized the power of the upcoming list. As the songs are primarily unheard, anyone who is recognizing the singer is doing it only based on their ability to identify the voice. This is the way to filter out the memorizers from the recognizers.
Soon the winner was crowned, and her ‘tab’ paid by the group. Julio thought of himself as a winner too, he constructed a solid foundation to effectively validate predictors. It is done by testing them on previously unseen situations.
The story illustrates an elementary point — ‘To effectively validate prediction methods, they should be tested on previously unseen data points.’
In Machine Learning lingo, this concept is called CROSS VALIDATION. The method suggests that the data available for the construction of the model should be split into two parts before using it for the model training.
Part 1 (usually randomly chosen 80% of the data) is used for TRAINING the model, and part 2 (remaining 20%) is kept aside during model training and used only to check the effectiveness of the constructed model, also called TESTING.
The concept is also known by its aptly named synonym, TRAIN TEST SPLIT.
~~ The Author intended this article to be read as Pre-Read before Machine Learning session. ~~
‘Ask Julio…’ is an ongoing work, and you can find all the other articles in the series here.
About ‘Ask Julio…’ series:
Ask Julio (pronounced like this) is a series attempting to explain Machine Learning to the ordinary; to those who in high school questioned what math does in real life; to those who have seen a code screen only in movies; and also to those who hear the words artificial intelligence and imagine Arnold Schwarzenegger from The Terminator.
As to speak to the 99%, Julio, the lead character of the article series, steps into the shoes of a lot of professions to experience how they learn things. During this journey, he asks a critical question, ‘How can anything be “learned”?’
This series of articles hopes to inspire people to learn Machine Learning by demonstrating one key point — it’s easier done than said.
About the Author:
Deepesh Wadhwani, a Mechanical Engineer, turned Data Scientist, has been associated with Data Analysis and Prediction for many years. He used to create Risk Prediction models for Wells Fargo Bank, and Pricing Models for Mitsubishi FUSO (Japan) before he pivoted his work on academics and started as Senior Faculty of Machine Learning and Data Science at International School of AI and Data Science. The cornerstone of his pedagogy is the effective translation of the Math (the primary language of ML algorithms) to run-of-the-mill arguments, which enables all in the audience to connect with the complex algorithms. In his classes, he uses a blend of logical conclusions and live demonstrations to reinforce the learnings.
