Guide to Simple Ensemble Learning Techniques

In one of our articles, we discussed the theoretical knowledge of ensemble learning methods, where we saw the working styles of different ensemble learning approaches. In simple words, ensemble learning can be concluded as the approach of making decisions based on the decisions of multiple machine learning methods. We saw in the article that there are different types of approaches, such as simple and advanced ensemble learning methods. So this article will let us know more about simple ensemble learning approaches using the below table of contents.

Table of Contents

• What do the simple techniques of ensemble learning mean?
• Max Voting/Majority Voting

1. Explanation
2. Implementation

• Averaging

1. Explanation
2. Implementation

• Weighted Averaging

1. Explanation
2. Implementation

What do the simple techniques of ensemble learning mean

When we look back at the last article, we can say that the simple techniques of ensemble learning refer to the basic approaches used to combine multiple individual models or learners to create a stronger and more accurate predictive model. Generally, these methods can be used as a foundation step in ensemble learning before exploring more advanced methods.

However, in many real-life use cases, we find these simple techniques as the best-fit solutions to the problem. For example, in many classification problems, the Majority Vating technique is an optimum solution because it not only improves the accuracy but also offers robustness, reduction of overfitting, compatibility with different models and many more things.

So before exploring the advanced ensemble learning techniques, it becomes important to get hands-on experience with the simple ensemble learning methods. So in later sections, we will look at the technicality of simple ensemble learning methods and how we can implement one using the Python programming language. The following simple ensemble learning methods will be explored.

• Max Voting
• Averaging
• Weighted Averaging

Let’s discuss them one by one.

Max Voting/Majority Voting

Explanation

Max or majority voting techniques refer to the same ensemble learning techniques as they are used to describe the process of combining predictions from multiple models in an ensemble and selecting the prediction with the highest number of votes or the majority of votes as the final prediction.

Technically in this method, each individual model in the ensemble makes its own prediction, and the prediction that occurs most frequently among the models is chosen as the ensemble’s final prediction. It is a simple yet one of the effective ways to aggregate predictions and obtain a consolidated decision from the ensemble. Let’s take a look at how we can work with this technique in Python.

Implementation

Here to look at the implementation of the majority voting ensemble learning method, we will use the popular breast cancer classification problem, which describes the possibility of breast cancer based on different symptoms and values. Let’s start the implementation by importing the data and necessary library.

from sklearn.datasets import load_breast_cancer

from sklearn.model_selection import train_test_split

from sklearn.ensemble import VotingClassifier

from sklearn.neighbors import KNeighborsClassifier

from sklearn.tree import DecisionTreeClassifier

from sklearn.svm import SVC

from sklearn.metrics import accuracy_score

Let’s load the dataset,

data = load_breast_cancer()

X, y = data.data, data.target

Let’s have some insights into the data.

print(“Feature Names: \n”, data.feature_names)

print(“Target Names: \n”, data.target_names)

print(“Data Shape: \n”, data.data.shape)

Output:

Here we can see the features and target values in the data as well as the shape of the data. Let’s take a look at the few samples of data.

print(“First few samples:”)

print(data.data[:5])

Output

Here we can see the samples we get with the data. Now we are required to split the data to train multiple models and test them.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Here we have split the data. Now we can define the individual classifier.

clf1 = KNeighborsClassifier(n_neighbors=5)

clf2 = DecisionTreeClassifier()

clf3 = SVC(probability=True)

Now the sklearn-provided VotingClassifier module will help us create the voting classifier with majority voting.

voting_clf = VotingClassifier(estimators=[(‘knn’, clf1), (‘dt’, clf2), (‘svm’, clf3)], voting=’hard’)

This voting classifier can be fit in the above-defined training data and make predictions.

voting_clf.fit(X_train, y_train)

y_pred = voting_clf.predict(X_test)

Let’s evaluate the performance of the model we made using the majority voting system.

accuracy = accuracy_score(y_test, y_pred)

print(“Accuracy:”, accuracy)

Output:

Here we have seen the complete implementation of the Majority Voting ensemble learning technique. Let’s move towards our second simple ensemble learning technique.

Averaging

Explanation

As discussed in the last article, in this method, the predictions from each individual model are averaged to obtain the final ensemble prediction. So here we say that the final result is made out from the equal contribution of each model, regardless of its performance or reliability. Technically the averaging process is typically performed by taking the mean or mode of the predictions, depending on whether it’s a regression or classification problem.

For example, in a classification problem, if three individual models predict class labels as [1, 2, 1], [2, 2, 2], and [1, 1, 2], respectively, the averaging ensemble would take the majority vote for each instance, resulting in the final prediction of [1, 2, 2]. Let’s take a look at the implementation of it using python.

Implementation

Here also, we will use the same breast cancer data, so here we will start the process by defining models.

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

from sklearn.linear_model import LogisticRegression

clf1 = RandomForestClassifier(random_state=42)

clf2 = GradientBoostingClassifier(random_state=42)

clf3 = LogisticRegression(random_state=42)

Here we have defined three different models. Now let’s train them using the training data.

clf1.fit(X_train, y_train)

clf2.fit(X_train, y_train)

clf3.fit(X_train, y_train)

Let’s make predictions from each model.

clf1_predictions = clf1.predict(X_test)

clf2_predictions = clf2.predict(X_test)

clf3_predictions = clf3.predict(X_test)

Now when we have predictions from each of the models, we need to combine the predictions using NumPy-provided averaging methods.

ensemble_predictions = np.column_stack((clf1_predictions, clf2_predictions, clf3_predictions))

averaged_predictions = np.mean(ensemble_predictions, axis=1).astype(int)

Let’s evaluate the performance of the ensemble.

ensemble_accuracy = accuracy_score(y_test, averaged_predictions)

print(“Ensemble Accuracy:”, ensemble_accuracy)

Output:

Here we have completed the implementation of averaging ensemble method. Now let’s discuss the last part of the article.

Weighted Averaging

Explanation

In this type of ensemble learning, we assign weight to each individual model’s prediction based on criteria like the importance or performance of the model. The weights can be determined based on various factors, such as the accuracy, confidence, or reliability of the model. To get the final results, individual models are multiplied by their respective weights, and the weighted average of these predictions is calculated.

As given in the above example, if the models are assigned weights of 0.4, 0.3, and 0.3, respectively, the weighted averaging ensemble would calculate the final prediction as follows:

[10.4 + 20.3 + 10.3, 20.4 + 20.3 + 10.3, 10.4 + 10.3 + 2*0.3] = [0.4 + 0.6 + 0.3, 0.8 + 0.6 + 0.3, 0.4 + 0.3 + 0.6] = [1.3, 1.7, 1.3]

In case the ensemble predictions are not integers, then the final prediction can be rounded to the nearest integer or adjusted based on specific requirements. Let’s take a look at its implementation.

Implementation

In the above-given codes, if we give the weights to the models, we can implement weighted averaging ensembling. So Let’s start by defining example weights for the model.

weights = [0.4, 0.3, 0.3]

Here again, we need to use the same Numpy provided function, but here we also need to define the weights. We can do this using the following codes.

ensemble_predictions = np.column_stack((clf1_predictions, clf2_predictions, clf3_predictions))

weighted_predictions = np.average(ensemble_predictions, axis=1, weights=weights).astype(int)

Let’s check the performance of the ensemble.

ensemble_accuracy = accuracy_score(y_test, weighted_predictions)

print(“Ensemble Accuracy:”, ensemble_accuracy)

Output:

Here we have completed the implementation of the weighted average ensemble technique.

Note: one this which is noticeable here is that the ensembling techniques such as random forest, Adaboost and gradient boosting come under the simple ensembling methods, but here we are not considering them because we conder them as a specific machine learning model, and here you can get the information about these models. By the nature of work, we can think of them as part of an ensemble learning methods group but are mostly used as a model.

Conclusion

Here in this guide, we have covered the simple ensemble learning methods, including majority voting, averaging and weighted averaging methods. Each of these methods has there unique style of working and can be used with different use cases. However, working with ensemble learning methods needs much calculation and power. Still, if the calculation and power are not a concern, then they are a good choice to achieve higher accuracy from the modelling.

About DSW

DSW, specializing in Artificial Intelligence and Data Science, provides platforms and solutions for leveraging data through AI and advanced analytics. With offices located in Mumbai, India, and Dublin, Ireland, the company serves a broad range of customers across the globe.

Our mission is to democratize AI and Data Science, empowering customers with informed decision-making. Through fostering the AI ecosystem with data-driven, open-source technology solutions, we aim to benefit businesses, customers, and stakeholders and make AI available for everyone.

Our flagship platform ‘UnifyAI’ aims to streamline the data engineering process, provide a unified pipeline, and integrate AI capabilities to support businesses in transitioning from experimentation to full-scale production, ultimately enhancing operational efficiency and driving growth.