Ensemble Methods for Machine Learning: AdaBoost

An implementation with Python

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could not be obtained from any of the constituent learning algorithms alone.

The idea of combining multiple algorithms was first developed by computer scientist and Professor Michael Kerns, who was wondering whether “weakly learnability is equivalent to strong learnability “. The goal was turning a weak algorithm, barely better than random guessing, into a strong learning algorithm. It turned out that, if we ask the weak algorithm to create a whole bunch of classifiers (all weak for definition), and then combine them all, what may figure out is a stronger classifier.

AdaBoost, which stays for ‘Adaptive Boosting’, is a machine learning meta-algorithm which can be used in conjunction with many other types of learning algorithms to improve performance.

In this article, I’m going to provide an idea of the maths behind Adaboost, plus I’ll provide an implementation in Python.

Intuition and Maths behind AdaBoost

Imagine we have our sample, divided into a training set and test set, and a bunch of classifiers. Each of them is trained on a random…