Understanding the Assumption of Independence in Regression
While performing regression, we deal with two assumptions in our data
1) Assumption of Addivity
2) Assumption of Linearity
Assumption 1 ask us to make sure that all features/attributes are independent. Assumption 2 states that there should be a linear relationship between response and predictors.
This blog is about Assumption 1 — Independence.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — -
To understand this assumption, first, let’s look at the solution of the regression and how we derived it.
In regression problems, we are given set of input features x1,x2..xn represented as matrix X for m observations in m by n dimension. For each observation, there is response value y1,y2..ym represented in vector y in m dimensions. [m = number of observations, n = number of input features]
Our task is to find a solution w such that Xw = y
I.E what linear combinations of columns of X will result in vector y.
Or w1x1 + w2x2+w3x3…+wnxn = y [w1,w2..wn are elements of vector w in n dimension and x1,x2..xn are vectors in m dimensions]
Most of the times we deal with the situation where there are a lot of observations related to the given features I.E matrix X is rectangular with m > n. This means that we have more equations to solve(m) and lesser variables available(n). In this case, Xw = y would have a solution only when y is in the same column space of X. I.E some linear combinations of columns of X can be equal to vector y. However, this is particularly not true in a real-world situation and hence, y will always lie outside of columns space of X.
As shown in the figure below, y is a vector in n dimensions and it is outside of the C(X) [columns space of X]
Now, to really solve our system, we would want to bring y in the same column space of X. How to do that — through projections
We will project y onto the column space of X. In Figure1, vector y is projected on X which is in a plane(representing column space of X) such that projection of y onto X is now equal to vector p.
and y = p + e
Where e is the perpendicular to the projection p and is also known as error vector.
Now, instead of solving Xw = y (which has zero solution)
we will now solve Xw = p ( where p is in now in same column space as that of X)
Or Xw = y-e
Now, in figure 1, we can see that the error vector e is perpendicular to the column space of X. So let’s assume for a moment, each column of matrix X is independent, so basis vector(columns which span columns space) will also be perpendicular to vector e so that
