What Makes Linear Regression Linear? With Examples
For most data scientists, statisticians, and even high schools the above picture is very familiar and not at all complicated. A line of best fit or a linear regression is a pretty simple model and has the following equation:
However, once say a squared term is introduced it no longer seems like a linear regression model.
While it may appear to go against the “linear” aspect of the model this is in fact still a linear regression model as counter-intuitive as it may seem.
The rule is that a linear regression model must be linear in the parameters, where the parameters are the beta values or coefficients. The most common form of linear regression (the first model given above) is confusing because it is linear in both the parameters and the predictors. Whereas the second equation is linear in only the parameters, but that is all that is required for linear regression.
Here, linear refers to a linear combination from linear algebra:
Below I’ve provided examples of linear and nonlinear models to be used a reference and hopefully make things more clear.
This model is considered nonlinear because of the second term which contains a nonlinear interaction between Beta1 and Beta2
The most commonly misclassified linear model, while Y is not a linear combination of x, it is a linear combination of Beta
If it seems like Y isn’t a linear combination of the predictors or the parameters you would be right. But, by simply redefining Beta 2 Squared to be Beta 3 and substituting in we then get a linear model! So yes the model as is not linear in the parameters, but it can be rewritten to be, so therefore it is linear! Try rewriting the example above to be linear, I bet you can’t!
This example may look a lot like the previous one, however, because of the exp() instead of a ()², it is nonlinear. This is because the last term cannot be broken up unlike in the previous example an there is no way to rewrite Y as a linear combination of Beta.
Example Credits: Professor Peter Frazier, Cornell ORIE
