Ensemble Learning- The heart of Machine learning

Ensemble learning Series…!!!

World of machine learning has full of different level of algorithm. but, still something is missing with data every time either it is bias in data or its variance or noise. When we use traditional machine learning algorithm method to train a model most of the case we find that we haven’t get the good accuracy and we find higher level of variance or loss in the result. That time we have to meta-learning which is nothing but ensemble of models. Here, we are going to introduce the ensemble machine learning Series in first phase then also we are trying to prepare this for deep learning.

Low Bias Low variance

Ensemble learning Series — (Happy Learning…!!!)

1. Ensemble Learning Relation With Bias and variance
2. Ensemble Learning- The heart of Machine learning
3. Bagging — Ensemble meta Algorithm for Reducing variance
4. Boosting- Ensemble meta Algorithm for Reducing bias
5. Stacking -Ensemble meta Algorithms for improve predictions
6. Ensemble learning impact on Deep learning

What is Ensemble Learning?

Ensemble methods combine several trees base algorithms to construct better predictive performance than a single tree base algorithm. The main principle behind the ensemble model is that a group of weak learners come together to form a strong learner, thus increasing the accuracy of the model. When we try to predict the target variable using any machine learning technique, the main causes of difference in actual and predicted values are noise, variance, and bias. Ensemble helps to reduce these factors (except noise, which is irreducible error).

Ensemble learning

Ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone…. Wikipedia

The fundamental principle of the ensemble model is that a group of weak learners come together to form a strong learner, which increases the accuracy of the model. When we try to predict the target variable by any machine learning technique, the main causes of the difference between the actual and predicted values are noise, variance and bias. The set reduces these factors (except noise, which is an irreducible error).

Here, we can see that if we will get the the variance, noise and bias in the raw data, image or any other format of the data. So,our model is going either under-fitting or over-fitting. This reason is create big impact on your model directly here the ensemble learning come in the picture. Training error and generalization error has gap which is represent as Generalization gap, which is show’s that model is under-fit or over-fit.

Imaging that Fable of blind men and elephant. All of the blind men had their own description of the elephant. Even though each of the description was true, it would have been better to come together and discuss their understanding before coming to final conclusion. This story perfectly describes the Ensemble learning method.

Ensemble learning Types

Ensemble Algorithms is divided into three main category basis of their use:

1. Sequential Ensemble learning (Boosting) :

• Boosting, is a machine learning ensemble meta-algorithm for principally reducing bias, and furthermore variance in supervised learning, and a group of machine learning algorithms that convert weak learner to string ones.
• Sequential ensemble methods where the base learners are generated sequentially.
• Example : Adaboost, Stochastic Gradient Boosting

2. Parallel Ensemble Learning(Bagging)

• Bagging, is a machine learning ensemble meta-algorithm intended to improve the strength and accuracy of machine learning algorithms used in classification and regression purpose. It additionally diminishes fluctuation of data(variance)and help to from over-fitting.
• Parallel ensemble methods where the base learners are generated in parallel
• Algorithms : Random Forest, Bagged Decision Trees, Extra Trees

3. Stacking & Blending

“Hands on Machine Learning with scikit-learn and Tensorflow.” (2017)

Stacking is a way of combining multiple models, that introduces the concept of a meta learner. It is less widely used than bagging and boosting. Unlike bagging and boosting, stacking may be (and normally is) used to combine models of different types. The procedure is as follows:

1. Split the training set into two disjoint sets.
2. Train several base learners on the first part.
3. Test the base learners on the second part.
4. Using the predictions from 3) as the inputs, and the correct responses as the outputs, train a higher level learner.

• Example : Voting Classifier

Blending is technique where we can do weighted averaging of final result.

References :

1. Ensemble Machine Learning Algorithms in Python with scikit-learn

2. Basic Ensemble Learning (Random Forest, AdaBoost, Gradient Boosting)- Step by Step Explained

3. https://stats.stackexchange.com/questions/350897/stacking-without-splitting-data

4. https://www.linkedin.com/pulse/do-you-want-learn-stacking-blending-ensembling-machine-soledad-galli/

5. https://en.wikipedia.org/wiki/Ensemble_learning