Linear and Polynomial Regression
Regression is a method of estimating a relationship from given data to depict nature of data set. This relationship can then be used for the forecasting future values or for computing if there exists a relationship among the various variables.
In this post, we’re going to look at Linear and Polynomial regression algorithms and their properties. We’ll soon find that many of them are biased to working well in certain types of situations and with certain types of data.
Linear Regression
In Linear Regression, there exists a linear relationship between the variables.The linear relationship amongst one response variable(dependent variable) and one regressor variable(independent variable) called as Simple linear regression and between one response variable and multiple regressor variable is called as Multivariate linear regression.
Y = a_1 * X_1 + b — — for simple Linear Regression
Y = a_1*X_1 + a_2*X_2 + a_3*X_3 ……. a_n*X_n + b — - for multivariate Linear Regression
It is quite easy to understand as we are simply weighting the importance of each feature variable X_n using the coefficient weights a_n. We determine these weights a_n and the bias b using a Stochastic Gradient Descent (SGD).The Linear Regression curve is of the form.
Linear Regression is very easy to understand and implement, but it is very sensitive to outliers.
Polynomial Regression
Polynomial Regression is a model which is used when the response variable is non-linear,it is rather a curve that fits into data points. General equation for Polynomial regression is of form:
y = b_0 + b_1 * (x_1)² + b_2 * (x_1)³ + … . . . . . + b_k * (x_1)^k
To solve the problem of polynomial regression, it can be converted multivariate linear regression with k independent variables:
y = b_0 + b_1 * x_1 + b_2 * x_2 + …………+b_k * x_k
where x_1 = x¹, x_2 = x² and so on.
Estimation of Parameters is done using least square method or Gradient descent method. using the least square method we get.
B = inverse((X*X’)) * X’ * Y
where B = vector of parameters, X = vector array of variables, Y = vector of observations.
Polynomial Regression computed on multiple regressor variables as Multiple Polynomial Regression. A second order multiple Polynomial expression can be expressed as: y = b_0 + b_1 * (x_1)+ b_2 * (x_2) + b_3 * (x_1)² + b_4 * (x_2)² + b_5*(x_1 * x_2)
Polynomial Regression is able to model non-linearly seperable data and is much more flexible than linear regression, but some of its disadvantages are we need some knowledge of data in order to select best exponents, and it is prone to over fitting if exponents are poorly selected.
Conclusion:
In this post, we have seen two of the most popular Regression techniques(Linear and Polynomial Regression) in detail. There are other Regression regularization methods( Like Lasso, Ridge and ElasticNet) which work well in case of high dimensionality and multicollinearity among the variables in the data set. I hope you enjoyed this post and Learned something new, feel free to give some claps.
