A Quick Overview to the Transfer Learning and it’s Significance in Real World Applications

In normal day-to-day life human beings use one domain of knowledge and apply it to another domain in order to solve a particular task in a much better way. More the similarity between the domains more easier it is to transfer and use the knowledge to get a much better output. In layman language there are many examples that occur in a normal human life such as:-

1. Knowing to ride a bicycle -> Learn to ride a bike
2. Know to play acoustic guitar -> Learn to play an electric guitar
3. Know the concepts of mathematics -> Learn the concepts of machine learning

In each of the above mentioned examples there is no need to learn anything from the beginning. Here there is only need to transfer the knowledge to from what one has learned from the past.

Generally the machine learning models are not trained to work by transferring the knowledge from one model to another. Knowledge here means the updated weights and biases obtained in each layers. Therefore, transfer learning is the idea to overcome this isolation and using the knowledge that is gained from one task to solve the related other tasks.

The main benefit of using transfer learning is that it can speed up the time to develop and train a model by again using the modules or parts of already developed models. These not only speed up the model training process but also enhance the results.

Andrew Ng, renowned professor and data scientist who has been associated with Stanford, Google and Coursera gave an amazing tutorial in NIPS 2016 which is called ‘Nuts and bolts of building AI applications using Deep Learning’ . In this lecture he mentioned that

“ Transfer Learning will be the next driver of ML success”

I recommend interested people to check out his interesting tutorial from Nuts and bolts of building AI applications using Deep Learning 2016.

Boom of Transfer Learning from 2016

From the graph discussed by Andrew Ng, it is clearly stated that the impact of Unsupervised algorithms and Reinforcement algorithms from 2016 has dropped to a great extent and the Transfer Learning has picked up the pace and is gaining speed. Although supervised learning is still much ahead of Transfer Learning but Transfer Learning will definitely overtake it in the coming years in terms of the importance.

Difference between Transfer Learning and Machine Learning

Machine Learning vs Transfer Learning

Traditional ML is termed as isolated because the knowledge is not accumulated. Moreover, it is a single task learning because the learning is done without considering the past knowledge into any other tasks. As from the above mentioned diagram it is inferred that the data is directly passed to the learning system or the training models and each model is trained without the knowledge transfer from other learning system.

In Transfer Learning, the concept is completely opposite from that of the Traditional ML because the learning of the new task depends on the previously learned tasks. With this act the accuracy of the output will be much better compared to its counterpart and the process will be enhanced. As seen from the above diagram it is clearly seen that the model that is trained transfers or shares knowledge (weights, features etc.) with the other learning system.

Demonstration of Transfer Learning vs Machine Learning

Let us understand the difference between Transfer Learning vs Traditional ML with the help of an example. Suppose one has a huge dataset comprising of images of different kinds of objects and that person has to classify whether the image of of a car or truck or some other vehicle. With the traditional ML the images has to pass through a Convolutional Neural Network which is going to be trained for the first time. Then the features will be extracted and after other CNN rounds the image classification will take place. On the other hand in transfer learning the pre-trained CNN model will be used which would have come from other Learning system and will give much better accurate results for the classification of images. Hence, in this manner Transfer Learning is getting much attention among the Machine Learning specialists.

Categories of Transfer Learning

So far we had discussed about the core meaning of Transfer Learning without diving deep into it’s mathematical concepts. These transfer learning strategies are dependent on the domain and availability of data. Therefore, the transfer learning techniques are broadly divided in to three categories based on the machine learning algorithms involved:-

1. Inductive Transfer Learning:- Here the source and target tasks vary from each other but the source and target domain are the same. AI algorithms help improve the target task by trying to use the source domain.
2. Unsupervised Transfer Learning:- This technique is very much similar to Inductive and the only difference is that the tasks are unsupervised in the target domain. Here also the tasks differ and the source and the target domain are similar. No labelled data is available here.
3. Transductive Transfer Learning:- This technique is completely different from the above mentioned two techniques. Here the source and target domains differ from each other while the tasks are somewhat similar. There is a lot of labelled data with the source domain whereas the target domain have no data.

Types of Transfer Learning Strategies

Now Transfer Learning can be applied to different settings but what to transfer is the main question. So, this answer lies in different kinds of approaches. Hence, I will be giving brief introduction about these approaches with which it would be very easy to understand the concepts.

1. Feature Representation Transfer:- Here the error rates and domain divergence are minimized to a great extent by identifying good feature representations that can be utilized from the source to target domains.
2. Relational-knowledge transfer:- This approach tries to to handle non-IID data, such as data that is not independent and identically distributed. Here each data point has a relationship with other data points.
3. Instance Transfer:- In most cases, the source domain data cannot be reused directly. Rather, there are certain instances from the source domain that can be reused along with target data to improve results. Reusing knowledge from the source domain to the target task is usually an ideal scenario.
4. Parameter Transfer:- This approach works on the assumption that the models for related tasks share some parameters or prior distribution of hyperparameters. Unlike multitask learning, where both the source and target tasks are learned simultaneously, for transfer learning, we may apply additional weightage to the loss of the target domain to improve overall performance.

Types of Transferable Components

Advantages of Transfer Learning

While one come down reading the article there has been several instances where I have mentioned the benefits of implementing Transfer Learning for creating cooler projects with quality output. So, here is some of the addition to those points:-

1. More robust models can be built which can perform wide variety of tasks.
2. Knowledge can be easily transferred from one model to other.
3. Sometimes the data don’t have labels so Transfer Learning can tackle that kind of problems.
4. Complex real-world problems can be solved with several constraints.

Drawbacks of Transfer Learning

No thing in the nature is perfect. Everything has some sort of drawback. We have come through a lot of positive sides of Transfer Learning but there are some disadvantages too which can cause some problems in your project if not taken care off. So it is important to keep them in mind while using Transfer Learning. There are mainly two types of Transfer Learning namely:-

1. Transfer Bounds:- It is very important in transfer learning to quantify the transfer. By doing this it affects the quality of transfer and its viability. This point is too broad to understand in this topic as it outside the scope of the article. For further reading you can refer to various research articles about the transfer learning like for instance that of Hassan Mahmud and their co-authors who gauged the amount for transfer and also that of Eaton and their co-authors presented a novel graph-based approach to measure knowledge transfer.
2. Negative Transfer:- There have been many instances where I had talked about the improvement in accuracy and performance with the help of transferring the weights, biases etc. to other learning systems. But sometimes there might be a degradation of performances. So, it is now easy to guess its definition. Negative Transfer learning refers to the drop in overall performance of the learning system during the transfer of knowledge from the source to target. It is very necessary to take care of this situation and requires careful investigation.

Conclusion

This concludes my first article with a brief coverage about Transfer Learning concept, difference from that of Traditional Machine Learning, it’s categories, the advantages and challenges of transfer learning. Transfer learning is definitely going to be one of the key drivers for machine learning and deep learning success in mainstream adoption in the industry. If you are reading this section, I am thankful for reading through my first article!

Resources

1. Chapter 11: Transfer Learning, Handbook of Research on Machine Learning Applications, 2009
2. Pan, S. J., & Yang, Q. (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345–1359.
3. A survey on transfer learning, 2010.
4. How to Improve Performance With Transfer Learning for Deep Learning Neural Networks