A Simple Introduction to Ensemble Learning

In one of our last articles, we discussed that random forest is an ensemble machine learning algorithm that predicts based on the combined predictions of multiple decision tree models. Since we found that ensembling multiple models or being an ensemble learning model is the main reason behind the success rate of any random forest model, this generates a curiosity to know more about the ensemble learning topic. So in this article, we will discuss the theoretical details of the ensemble machine learning method. We will cover the following important points about it:

Table of content

1. What is Ensemble Machine Learning
2. Ensemble learning Techniques

Simple Techniques

1. Max Voting
2. Averaging
3. Weighted Averaging
4. Advanced Techniques
5. Stacking
6. Blending
7. Bagging
8. Boosting

What is Ensemble Machine Learning?

As discussed above, ensemble learning is an approach that comes under supervised machine learning, where we use the combined results of several supervised learning models. So let’s try to understand it more using an example. Let’s say a person has written an article on an interesting topic, and he wants to know the preliminary feedback before publishing it. So he thinks about the following possible ways.

• Ask a friend to rate the article: In this way, there is a huge possibility that a friend of his gave a nice rating for a horrible work because he didn’t want to break the heart of his friend.
• Ask five friends to rate the article: This way, he got the proper idea of the article because some of them chose to give an honest rating to him. But there is also a possibility that the people are not subject matter experts on the topic of his article.
• Ask 50 people to rate the article: Here, he has included all his friends and some strangers to give feedback and got more generalized and diversified feedback. This approach can be the best of all the approaches he chose to get feedback for his work.

Here we can infer that if the person uses a diverse group of people to get the feedback, he will get better results as compared to a nondiverse group or individuals. This way can also be applied in machine learning, where a set of multiple models can give better results than a single model. This diversification in machine learning can be achieved using ensemble learning models.

Here we got a basic understanding of the ensemble learning technique. Let’s take a look at the various techniques using which we can perform ensemble learning.

Ensemble Learning Techniques

The following are techniques to perform ensemble learning:

Simple techniques

• Max Voting: we generally use this method for solving classification problems. Using each data point, multiple models give their outcome, and this outcome is considered as the vote. Using the majority vote-winning technique, we reach the final result.

Suppose there in the above example, the person chose to follow the second way and asked five friends to give feedback. If 4 or 3 out of 5 are on the positive side, which means they give more than 2 rating points, then the person will make the article published. The result of this structure of ensemble learning will take like as follows:

• Friend 1 = 3
• Friend 2 = 4
• Friend 3 = 5
• Friend 4 = 2
• Friend 5 = 3
• Averaging: Just like in the max voting system, here also, all the models take each data point to make predictions, but we consider the final result as the average of results from all the models. The averaging method is mostly applied in regression problems.

By the above-given results, we can calculate the final result in the following way:

(3 + 4 + 5 + 2 + 3)/5 = 3.4

• Weighted Average: This technique of ensemble learning is different from the other methods because we use the importance of each model as the weight, and the final result comes out from the weighted average method.

Let’s say 2 out of 5 friends are given more importance in the above example, and because of this weight or importance, the final result will be as follows.

(3 + 4 + 5 + 2 + 3)/5 = 3.4

Advanced techniques

• Stacking: if the above-discussed methods can be considered basic ensemble learning methods, then methods after this can be considered advanced ensemble learning. Stacking is a method where several learners are attached one after another. Decision tree, KNN and SVM algorithms can be considered examples of base models used in stacking learning. The following steps a stacked ensemble learning model takes to give final results:

1. Splits the training set into n parts.

1. A base model learns from the nine parts and predicts the 10th part of the data.

1. The base model gets fitted on all parts of the training data.
2. Using the trained model, we calculate the predictions using the test data.

1. Steps 2, 3 and 4 are repeated using different models like KNN or SVM, and that also gives predictions but different from the first cycle.

1. Predictions based on train data are considered as the feature to build a new model.

1. The final or new model gives the final prediction based on the test set.

• Blending: Blending is almost similar to the stacking approach, but the use of validation data makes it different, unlike the stacking method. It gives the predictions from the validation set only. This validation set and prediction were made using the validation set used to run on the test set. The steps this method uses are as follows:

1. The whole data is split into three sunsets: train, test and validation.
2. Models are trained on the training set.
3. A validation set and a train set are used to make the prediction.
4. Validation set and predictions made using validation set used as a feature to train a new model.
5. A new model and test data are used to make the final prediction.

• Bagging: Bagging is an advanced form of ensemble learning where it uses multiple models to give their individual results on a sub-part of data. By combining these results gives a final outcome. Since multiple models have a high chance of giving the same results while inputs are similar, bootstrapping comes into the picture to fail this condition. It helps create various subsets of whole data and then trains multiple models on those subsets. The below picture is an illustration of the bagging technique.

The following steps this technique uses to learn from the data:

1. Subsets are created from the original data.
2. A base model is assigned to learn from each subset.
3. The final prediction comes out as the combined result from all the models.

• Boosting: Boosting ensemble learning can be considered a sequential learning method, where each subsequent model tries to correct the errors from the previous model. It means that the outcomes of a succeeding model depend on the outcome of the previous model. The following steps this technique takes to complete the modelling:

1. Creation of a subset from the main data.
2. At the initial stages, all the data points have similar weightage.
3. A base model gets trained on the subset and gives predictions using the whole data.
4. Errors are calculated using the initial model’s original value and predicted value.
5. Incorrectly predicted data points take higher weights.
6. Again a base model is used to get trained and give predictions on the dataset.
7. The process from steps 3 to 6 is repeated until the final learner doesn’t occur.

Final words

Here in the article, we have discussed the basic introduction of ensemble machine learning. Using an example, we tried to understand how it works and learn about the different ensemble learning techniques, such as max voting, averaging, bagging and boosting. In our next articles, we will discuss the models based on ensemble learning techniques.

About DSW

Data Science Wizards (DSW) is an Artificial Intelligence and Data Science start-up that primarily offers platforms, solutions, and services for making use of data as a strategy through AI and data analytics solutions and consulting services to help enterprises in data-driven decisions.

DSW’s flagship platform UnifyAI is an end-to-end AI-enabled platform for enterprise customers to build, deploy, manage, and publish their AI models. UnifyAI helps you to build your business use case by leveraging AI capabilities and improving analytics outcomes.

Connect us at contact@datasciencewizards.ai and visit us at www.datasciencewizards.ai