Understanding the AdaBoost Algorithm

6 min readJul 7, 2023

People related to the field of data science and machine learning already know about the term boosting, and as discussed in the series of articles sharing knowledge about ensemble learning in machine learning, ensemble learning is a course where we use multiple methods to build a final and strong model and boosting is part of this course which makes a difference from traditional ensemble methods by assigning varying weights to each learner, focusing more on the instances that were previously misclassified. There are multiple boosting algorithms like AdaBoost, Gradient Boost, LightGBM and CatBoost.

In this article, we will be discussing the AdaBoost algorithm, which is one of the earliest boosting algorithms which came into real-life adaption for solving problems and is based on the idea of boosting (combining multiple “weak classifiers” into a single “strong classifier.” ). Using the following table of contents, we will get to know the fundamentals of the AdaBoost algorithm.

What is the AdaBoost Algorithm?
How Does The AdaBoost Work?
Code Implementation
Advantages and Disadvantages of AdaBoost

What is the AdaBoost Algorithm?

In the space of machine learning models, there are multiple options we can choose from, and AdaBoost is one of them. It is a part of the advanced ensemble learning models family and follows the way that boosting learning algorithm follows that we have discussed in our articles.

Talking about the main idea behind the AdaBoost algorithm, we find that it iteratively trains a sequence of weak classifiers on different subsets of the training data. During each iteration, the algorithm assigns higher weights to the misclassified samples from the previous iteration, thereby focusing on the more challenging examples. This process allows the subsequent weak classifiers to pay more attention to the previously misclassified samples and improve their performance.

In a nutshell, we can say that adaptive boosting is a way to reduce the error of any machine-learning algorithm, which work by aligning many weak machine-learning models into one strong machine-learning model.

How Does The AdaBoost Work?

We can understand the working of the AdaBoost algorithm in step by step manner as going deep into the work, we can see there are multiple basic steps which this algorithm follows. Let’s take a look at these steps.

Step 1: When the algorithm is given data, it starts by Assigning equal weights to all training examples in the dataset. These weights represent the importance of each sample during the training process.

Step 2: Here, this algorithm iterates with a few algorithms for a specified number of iterations (or until a stopping criterion is met). The algorithm trains a weak classifier on the training data. Here the weak classifier can be considered a model that performs slightly better than random guessing, such as a decision stump (a one-level decision tree).

Step 3: During each iteration, the algorithm trains the weak classifier on given training data with the current sample weights. The weak classifier aims to minimize the classification error, weighted by the sample weights.

Step 4: After training the weak classifier, the algorithm calculates classifier weight based on the errors of the weak classifier. A weak classifier with a lower error receives a higher weight.

Step 4: Once the calculation of weight completes, the algorithm updates sample weights, and the algorithm gives assigns higher weights to misclassified examples so that more importance in subsequent iterations can be given.

Step 5: After updating the sample weights, they are normalized so that they sum up to 1 and Combine the predictions of all weak classifiers using a weighted majority vote. The weights of the weak classifiers are considered when making the final prediction.

Step 6: Finally, Steps 2–5 are repeated for the specified number of iterations (or until the stopping criterion is met), with the sample weights updated at each iteration. The final prediction is obtained by aggregating the predictions of all weak classifiers based on their weights.

The below pseudocode can be helpful in understanding the working of the AdaBoost algorithm.

Initialize sample weights for each training example
For each iteration:
Train a weak classifier using the current sample weights
Calculate the error of the weak classifier
Calculate the weight of the weak classifier based on the error
Update the sample weights based on the weak classifier's performance
Normalize the sample weights
End the iterations
Combine the weak classifiers using a weighted majority vote.

Now that when we know the working of this algorithm, let’s take a look at the code examples using which we can take benefits of AdaBoost for our use-cases.

Code Implementation

Let’s start by importing important libraries which we will require in the process.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import AdaBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression

Here we have imported the libraries. To complete the purpose, we will use the breast cancer dataset given under sklearn’s package dataset. Let’s import the data.

# Load the Breast Cancer dataset
data = load_breast_cancer()
X = data.data
y = data.target

Let’s take a look at the basic information of the data.

print('name of the Features \n', data.feature_names)
print('name of the classes \n', data.target_names)
print('name of the classes \n', data.data.shape)

Output:

Here we can see the basic information about the data. Now let’s move towards defining models.

In this article, we are going to use two different models, one is logistic regression, and the other is the AdaBoost model so that we can compare the results between them.

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
#Logistic Regression Model
log_reg = LogisticRegression()
#AdaBoost classifier
adaboost = AdaBoostClassifier(n_estimators=50, random_state=42)

Let’s train the model using the fit function.

# Train the classifiers
log_reg.fit(X_train, y_train)
adaboost.fit(X_train, y_train)

Making predictions using the trained model.

# Make predictions on the test set
y_pred_1 = adaboost.predict(X_test)
y_pred_2 = log_reg.predict(X_test)
Let's check the accuracy of both models.
# Calculate the accuracy of the model
accuracy_1 = accuracy_score(y_test, y_pred_2)
accuracy_2 = accuracy_score(y_test, y_pred_1)
print("Accuracy of logistic Regression:", accuracy_1)
print("Accuracy of AdaBoost:", accuracy_2)

Output:

Here we can see that the accuracy of the AdaBoost model s greater than a single logistic regression model.

Advantages and Disadvantages of AdaBoost

In the above sections, we have discussed the working method of the AdaBosst algorithm, and now we need to discuss its advantages and disadvantages so that we can easily understand when and where we can use it. Let’s take a look at the below table.

Final Words

In this article, we have discussed the AdaBoost algorithm, which is a member of ensemble learning models and, specifically, a member of boosting algorithms. Since it was one of the earliest implementations of boosting algorithms, it becomes important to understand the pattern which it uses in its working so that the advanced boosting methods can become easy to understand. Here we discussed the definition of the AdaBoost algorithm and how it works, and we saw a code implementation of AdaBoost using the Python programming language.

In the end, we have discussed the advantages and disadvantages of the AdaBoost algorithm and using this information, we can make a clear decision before utilising the AdaBoost algorithm for any of our problems.

About Us

DSW, specializing in Artificial Intelligence and Data Science, provides platforms and solutions for leveraging data through AI and advanced analytics. With offices located in Mumbai, India, and Dublin, Ireland, the company serves a broad range of customers across the globe.

Our mission is to democratize AI and Data Science, empowering customers with informed decision-making. Through fostering the AI ecosystem with data-driven, open-source technology solutions, we aim to benefit businesses, customers, and stakeholders and make AI available for everyone.

Our flagship platform ‘UnifyAI’ aims to streamline the data engineering process, provide a unified pipeline, and integrate AI capabilities to support businesses in transitioning from experimentation to full-scale production, ultimately enhancing operational efficiency and driving growth