Multiple Regression without iteration (Mathematical Intuition)

Let’s first understand what multiple regression is. Consider an example such as x = [1,2,3,4,5] and y = [1,2,3,4,5]. (In this equation, x is an independent variable and y is a dependent variable.) Since there is only one independent variable present in this instance, multiple regression data typically contains multiple independent variables, as shown in the example below.

As we all know, the standard line equation is y=mX + c, but it’s not possible to fit the above points with a line because there are three parameters (X1, X2, and Y), so a 3D plane is used to fit the above points or any multiple regression problems. The standard plane equation to fit the multiple regression problems is y = b0 + X1*b1 + X2*b2 + X3*b3.….. so on.

Let’s derive the formulas for b0, b1 ,b2…. . Let’s make use of cost function(Which helps to note down the loss) ,i.e.,

Substitute the y= b0 + X1*b1 + X2*b2 + X3*b3.….. , in cost function.

If you consider the above example, X1(0) is basically 32. Similarly, X2(0) is 10. Now transform the above equation in matrix form, i.e.,

Since we know that matrix A2 can be written as A.A., we can write A.A. in linear transformations as ATA (i.e., A(Transpose)A). Using this in our equation, we get

Neglect the 1/n term because it doesn’t affect the equation, and take the derivative of cost function and equate it to zero.

Therefore, this equation is known as “NORMAL EQUATION”.

As a result, b is a 1Xn matrix with b0, b1,... bn values (where n is the number of independent variables plus 1).

The inverse of XTX in b is sometimes non-invertible (the determinant is zero). To avoid this remove some features or use some regularization techniques.

This method is not preferred if the number of independent variables is greater than 1000 because it becomes complex to find the inverse of a such large entity.