What Is Transfer Learning
A Quick Introduction
"Transfer learning". This concept has revolutionized deep learning, but what is it actually ?
If you want to listen to this story on the move then this story is available as a podcast on “SimpleAI”. Click on play below.
The “Why” Of Transfer Learning
Over the years BIG DATA has partly enabled us to achieve long strides in deep learning.
Yet there are some domains where getting big data for a problem statement becomes impossible. Moreover, having access to big data is almost always possible only for big companies and institutions with big money.
You and I don’t have those kind of resources.
The “How” Of Transfer Learning
To understand an example I will use an example.
I speak three different languages and one of the languages that I speak is “Odia” which happens to be my native language.
The Odia language has a rich history and is in existence for thousands of years. There is a link to the Wikipedia article at the end of this post incase you wan to know more about the Odia language
This is a low resource language. What it means is that the documentation, translation and data for this language is very rare.
In simple terms there is not much curated dataset available based on the language.
What if I wanted to classify some documents in the Odia language using a deep learning model?
For this I would need huge data to train a neural network. But my native language being a low resource doesn’t has a big enough dataset.
So, what should I do now?
Well! it seems like I can use a neural network which was earlier trained on a dataset in my native language and then use the knowledge learned by that model in the tasks further down the chain.
This is nothing but transfer Learning on a very high level.
To dive a little deeper into the process I would follow the following steps-->
- Take a neural network.
- Train it on Wikipedia articles written in Odiya.
• Now that he Neural network is trained on the Odia language corpus. It’s time to fine-tune it on the target task.
First of all Remove the head of the neural network but keep it's body. The body is where the neural network stores the knowledge like the structure of the language.
• Then I would retrain the body of the neural network on a target dataset in Odia and then use this new network for some other tasks like classification.
This is a typical transfer learning pipeline.
This is a very high level explanation of how transfer learning is done but there are lots more which I am not listing here. Yet, I have provided links to some interesting articles on this topic.
Transfer learning has enabled the reuse of deep learning models across domains and tasks. This is one of the catalyst towards democratizing Deep Learning.
Conclusion
Let me know how you felt about this story, what you learned and what I missed by a tweet, email or commenting on this story.
End Notes
Creating a content for you requires me to do a lot of research, planning, writing and rewriting . This is important because I want to deliver practical content to you without any fluff.
If you like my work and my content and want to support me then the following are the ways to show your support →
Support me
- If you like my work then click on this link to Buy me a coffee.
- Buy my deep learning course at udemy. Just click on the course link in the show notes and get awesome discount on my deep learning course.
- Subscribe to my publication and share it across so that more people can discover it.
- Subscribe and share my podcast “SimpleAI” on google podcast or any other podcast player of your choice. Don’t forget to give it a 5 star.
- Subscribe to my newsletter.
Recommended Reading List
- What Is Transfer Learning?- By Scott Martin, Published in Nvidia AI Blog.
- A Comprehensive Hands-on Guide to Transfer Learning with Real-World Applications in Deep Learning- By Dipanjan Sarkar, Published in “Towards Data Science”
