Member-only story
Gradient Boosting in Python from Scratch
Coding and explaining in depth the very popular and competition-winning gradient boosting algorithm using Python
The aim of this article is to explain every bit of the popular and oftentimes mysterious gradient boosting algorithm using Python code and visualizations. Gradient boosting is the key part of such competition-winning algorithms as CAT boost, ADA boost or XGBOOST thus knowing what is boosting, what is the gradient and how the two are linked in creating an algorithm is a must for any modern machine learning practitioner.
The implementation and animations of gradient boosting for regression in Python can be accessed in my repo: https://github.com/Eligijus112/gradient-boosting
The main picture of the article depicts the process of evolution and how, over a long period of time, the beak size of a bird species adapts to its surroundings. https://en.wikipedia.org/wiki/Darwin%27s_finches
Just like animals adapt in various ways given new facts in their habitats, so do machine learning algorithms adapt to the data environment we put them in. The main idea behind the gradient boosting algorithm is that the main engine of it is a low accuracy and simple algorithm which learns from its own previous mistakes.
