Why XGBoost model is better than neural network once it comes to regression problem
XGBoost is quite popular nowadays in Machine Learning since it has nailed the Top 3 in Kaggle competition not just once but twice. XGBoost features gradient boosting algorithm which can be used for classification and regression problems alike. However, in last couple of months XGBoost became very promising contender when solving regression problem as well and even outperformed neural networks trained on L2 regularization .
On the other hand XGBoost is specialized only on one kind of problem — regression. XGBoost tries to fit data with a linear function to get best possible coefficients which minimizes loss function (it’s called Ordinary Least Square method). In this sense XGboost treats regression problem exactly like classical statistical problems.
What is XGBoost model? and what is the form of it?
XGBoost model starts with a simple decision tree and it’s trained X times. X is number of trees and X can be passed as an input parameter (it’s called n_estimators by default). Then XGBoost tries to find best possible coefficient for linear combination of features — each tree gives one coefficient which is used for linear formula.
An example tree:
