Deep Learning: Where did it Come From?
Deep Learning is one of the most popular buzzword thrown around in the tech community today. Everyone talks about Deep Learning, Machine Learning, Artificial Intelligence and how it’s the future of technology. The problem is — many people don’t know what they really are.
Artificial Intelligence is a Field of Study
I like to think of Artificial Intelligence as Engineering. Well, in Engineering, we have several sub-fields like: Civil engineering, Mechanical engineering, Chemical engineering, Software engineering, Electrical engineering, etc… So also does Artificial Intelligence have its own sub-fields: Computer Vision, Natural Language Processing, Expert Systems, Robotics, Optimization, Machine Learning, Data Science, Fuzzy Logic, etc…
Machine Learning
One of the sub-fields of Artificial Intelligence is Machine Learning. In Machine Learning, instead of writing a program to solve a problem, we write a program to learn to solve a problem from examples. And it’s in this process of learning that the system gets better over time. Machine learning has 3 major categories: Supervised Learning, Unsupervised Learning & Reinforcement Learning.
Why is it important to learn how to solve a problem from examples? Well I’ll show you an example. Let’s say we are given a task to write a program to recognize a cat in a picture. How do we do that? Well, one thing we can do is write some rules as to what a cat will look like. We can say to a computer, in Mathematical language that: a cat has a round face with two pointy ears, a long tail and a chubby body. Awesome! Well, this seems to work well enough for this cat (on the left), right?
Okay, what if we’re now given this cat (on the right) here? Our rules starts to break. But that’s fine, we can always update our rules and have more complex description of a cat. But then we’re given these other pictures of a cat who is hidden from viewport, and all these silly cats. Now what do we do? Notice that our program which we’ve spent forever writing rules for only works for cats and nothing else. If we’re told to recognize a dog, we’ll have to start writing the rules all over again. Yikes!
But what if we have a system that takes in a picture of a cat, and gives us back a score that “I’m 92.6% sure that this is a cat”? Wouldn’t that be great?! Well this is exactly the reason for Machine Learning. We give the machine lots of pictures of a cat and that of a dog, and then the machine starts to write those rules that we would have otherwise written ourselves. Now this system can be used for many other purposes. We could just swipe out the picture of a cat with that of a dog, or a car or an antelope.
Machine Learning & The Neural Network
There are tons of Machine Learning models out there that are used for performing these tasks (i.e learning from examples): Linear Regression, Support Vector Machines, Decision Trees, Logistic Regression, K-Nearest Neighbors, Naïve Bayes, Random Forest, K-Means, Mean Shift, just to name a few. But out of all these models there’s one that stands out — Neural Network. It was built roughly on the principles of how the brain works.
A shallow neural network has 3 layers: Input layer, a hidden layer & output layer. Each layer is made up of series of neurons (which is a fancy term for Linear Transformation i.e. y = Wx + b). What the input layer does is to take in some input data (e.g. picture of a cat in our case), perform some transformation and give the result of those transformation to the next layer (hidden layer). The hidden layer also does it's own transformation and sends the result to the output layer –which could say I'm 94.7% sure that this is a cat!
This obviously is a very simplified version of what the neural network is. Anyways, the most interesting thing is what happens when you increase the number of hidden layers in the network.
Let’s Go Deeper!
This is what gave birth to the sub-field of Machine Learning called Deep Learning. Deep Learning is all about having many hidden layers and orchestrated in all sorts of ways that it outperforms all other vanilla Machine Learning models across a wide range of tasks. With more hidden layers, comes the need of more computing resources to train these deep networks & also lots of data for the network to work with.
Therefore: when you have a deep neural network architecture + big data + sh#t loads of compute power –This is Deep Learning.
Today, there have been many deep neural network architectures –some popular ones are AutoEncoders, Generative Adversarial Network, Residual Network, Transformer, Inception, VGG16 and many more…
