Introduction
We humans not only learn things but can transfer that knowledge across tasks. We acquire some knowledge while learning a task, we use this knowledge to solve related tasks. It is easy for us to solve a task if that task was more related as we can apply the prior knowledge gained from other tasks.
Some examples in our daily life would be,
- Know how to play a keyboard. . . . Learn how to play a piano
- Know how to ride a bicycle. . . . Learn how to ride a motorbike
In the above scenarios, we don’t start learning from the scratch when we try to learn new skill. We utilise and leverage our knowledge from the past.
From the word “Transfer learning” we intuitively get to know the meaning as transferring something that is learned. Conventional neural network models and deep learning techniques have been traditionally designed to work well in isolation. These models need to be changed from the scratch whenever the feature space distribution is changed. Transfer learning on the other hand is the idea to overcome this rebuilding by learning and utilising knowledge acquired in the past to solve related tasks. Few things discussed in this article regarding transfer learning are it’s concepts, future scope and real world applications.
Transfer learning is a concept in machine learning in which the knowledge gained while solving a problem is stored and that knowledge is used to solve a related but different problem. TL provides a jump start for a new model as it is is using pre-trained models used for other tasks. History of psychological literature somehow related to this field of research.
Formal definition
Domain and tasks are the things that transfer learning is defined on. Domain, D, feature space and marginal probability together in a tuple.
Here xᵢ denotes specific vector. A task, T, label space,γ , and objective function, η together in a tuple.
As you are now familiar with these definitions and representations, now we can look at a definition given by a data scientist as, taken from an article(by SebastianRuder).
This example will you help you understand the difference between domain and task better.
Motivation
As we discussed earlier, humans don’t learn everything from ground up, we transfer and leverage knowledge gained from learnt domains to solve newer domains or tasks. Andrew Ng, very well known data scientist and professor, gave an lecture in ‘Nuts and bolts of building AI applications using Deep Learning’, where he proposed-
After Supervised learning, Transfer learning has the potential to become the next driver of ML.
TL is not a concept that popped up in the 2010s.
The key motivation that drives deep learning towards TL is that the models require large datasets, and getting huge amount of labelled data for supervised learning, when time and efforts are main criteria while labelling data points, it might be difficult. A simple understandable example would be, ImageNet Dataset, which consists of millions and millions of image pertaining to certain categories. Collecting such kind of dataset for every domain is very difficult. Most of the deep learning algorithms restricted to a particular domain or even to a particular task. These DL models have a very high accuracy and can break all benchmarks and end up suffering a huge loss in performance when used on new tasks which even might be similar to this trained one. These are the things that motivated transfer learning, going beyond specific domains and tasks with the knowledge of pre-trained models to solve new problems.
Understanding Transfer Learning
Main thing to remember is transfer learning is not a new thing that is specific to deep learning. There is a clear difference in traditional way to build models and using a method like transfer learning.
Traditional method is an isolated approach that solely concentrate on specific tasks. No knowledge is stored which can be used for similar models. In Transfer learning, we utilise and leverage knowledge from past trained models to solve related models much faster and better.
Applications and Examples
Deep learning is the most important and that category which uses transfer learning very successfully.
- Transfer learning for NLP(Natural language processing): When dealing with textual data there are many challenges that needs to be overcome using DL and ML. Data is transformed or vectorized using various different techniques. Embeddings, such as FastText and Word2Vec, have been trained on various different datasets. These are utilised in tasks like sentiment analysis and document classification, by transferring knowledge across tasks.
- Transfer learning for Speech/Audio: Deep learning is extensively used in audio domain. To improve speech recognition of other languages Automatic Speech Recognition(ASR) mainly developed for English are being used and has shown better performance too. Automated speech identification also uses transfer learning.
- Transfer learning in Bayesian networks and Markov logic networks
- Transfer learning in digit recognition
- Transfer learning in cancer subtype discovery
Examples
- Image classification using Inception V3
- VR enabled limitation learning
- Meta learning
- Transfer learning with warm restart
When to use Transfer Learning
Transfer learning is basically an optimisation algorithm that saves time or enhances performance.
Three major benefits using Transfer learning:
- Higher slope. The rate of improvement during training is steeper.
- Higher asymptote. Trained model’s convergence skill is better than would be.
- Higher state. Initial skill would be higher.
It is an approach when you have abundant data to build model for that task and can use it for similar task or you can use a pre-trained model for your task.
On some problems when you don’t have enough data to build a model, you can use transfer learning to build that model. There might be some cases which cannot be solved without using transfer learning.
Transfer Learning Advantages
By know you might have understood some of the advantages using transfer learning. Typically TL helps us to build more robust models which can perform various different tasks.
- Simplifies complex real-world problems with some constraints
- Helpful with data having little or almost no labeled data
- Transferring knowledge across models based on domains and tasks becomes easy
- Way towards achieving Artificial General Intelligence(AGI) in the future.
Transfer Learning Challenges
- Negative Transfer: Until now, we have seen cases that improve performance of the model due to transfer learning. There are some cases that doesn’t have any impact on the performance, in fact, some may decrease the performance of the model too, this is negative transfer.
- Transfer Bounds: The amount of data that is transferred during transfer learning also needs to be taken care as this might even effect the model’s performance.
The Future
As Machine Learning continues to grow and adapt various new techniques. Transfer Learning and Meta Learning as models will grow and get more integrated and automated. Usage of exiting frameworks and models training themselves can create efficiencies not even imagined in the past. These enhancements and techniques will accelerate the potential of Deep Learning and Machine Learning and move programmers to use more deep neural networks that write their own code and enhance their own behaviours.
Advanced companies already started using deep neural networks to run their businesses. These companies have built such models that in many cases self-train as data flows in, and can learn, adapt, improve, make decisions, can provide seamless and fluid experience and do many more tasks.
Summary
In this article you have learned how to use transfer learning to improve the performance and to speed up training of your deep learning model.
