Ever wondered how to Calculate the coefficients of Simple Linear Regression? | Opening the Blackbox

In this post I tried describing how a simple linear regression works and used Ordinary least squares to minimise the cost function of Mean Squared Error (MSE). After doing that we got the values for β0 and β1

Expressions for coefficients of Simple Linear Regression

In this post we would go through the derivation of the above values using ordinary least squares to minimise the cost function i.e. MSE

Since typing equations is a pain on computer, I would be using pen and paper in most of my posts going forward.

Lets re-look at our cost function :

Eq 1 : Cost Function for OLS Simple Liner Regression

Now replacing the value of ŷ and re-writing the equation, we would get

Eq 2 : Cost Function for OLS Simple Liner Regression

Now, we need to find the value of β0 and β1 that would minimise the value of this function. To minimise it, we would calculate partial derivative of the function w.r.t β0 and β1 .

Lets start by calculating partial derivative of cost function w.r.t β0

Eq 3 : partial derivative of cost function w.r.t β0

Lets now calculate the partial derivative of cost function w.r.t β1

Eq 4 : Partial derivative of cost function w.r.t β1

To summarise, the partial derivatives of β0 and β1 that we derived above, we got the following results —

Eq 5 : Partial derivative of cost function w.r.t β0 & β1

Now to find the value of β0 & β1 we need to equate the above partial derivatives to 0.

To start with lets equate the partial derivative of β0 to zero and that would give us the equation for β0 that would minimise the cost function.

Eq 6 : Finding β0

Lets now equate the partial derivative of β1 to zero and see what we get.

Eq 7 : Finding β1

In the above equation (Eq 7), lets replace the value of β0 from Eq 6. On doing that we would get the below :

Eq 8: Finding β1

Thats great, now we have calculated the value for β0 and β1 that would help us minimise the cost function. And we know the math behind it as well.

The results from this is what we use while building a linear regression. I have written a post on how to build your first linear regression model from scratch here.