What is Gradient Tree Boosting? — Part 1

As most posts I upload discuss aspects of DEEP learning, I thought it would be nice to freshen up a little with the strong concept of gradient tree boosting.

Gradient Tree Boosting, also known as Gradient Boosted Decision Trees, combines the strengths of decision trees and boosting algorithms to create a formidable predictive modeling tool. Iteratively refining a set of weak decision trees through gradient-based optimization achieves remarkable predictive accuracy and the ability to capture complex relationships within the data.

Tree Boosting

we consider K Tree models that are used in an ensemble to generate a prediction, that is the sum of predictions from each tree. Denoting our final model as f, the following figure shows how f(boy)=2.9

More formally, for a dataset with n samples and m features

dataset

sample, target and prediction

function and structure of the tree

epresents the structure of the tree (the decision rules that the tree is made up of), that is, it takes an input and assigns it to some leaf, and wq(x)​ is the leaf weights that are provided if the input is x and the structure is q.
Each of the T leaves contains a continuous score which is wi​, and the final prediction is the sum of the scores in the leaves. This means we can write eq.1 as

prediction in terms of wq

Regularized Learning Objective

regularization term

Up next…

Though our objective function is well-defined, there are some tricks that can help us optimize it. In the next section, we will deal with the Gradient term in Gradient Tree Boosting