Simple Linear Regression for Machine Learning made easy with Ordinary Least Square [OLS] Method
Hello Everyone!
I am super excited to be writing another article after a long time since my previous article was published.
A Simple Linear Regression [SLR] is basically this formula:
which is spelled as y equals b zero plus b one times x one. I am sure you have seen this formula in your high school which was a part of drawing a line or sloped line in a x-y axis. Let’s move a step ahead and understand what each of these variables or coefficients mean in detail.
What does y signify in the equation?
From the above equation, y is the dependent variable (DV), It is a variable which is trying to explain something, For Example:
Hypothetically speaking Salary of an employee depends on the years of experience. In this case y that is the salary of an employee would be the dependent variable, since it is dependent on the years of experience.
or let’s take another example where the marks scored by the student depends upon the number of hours spent for studying, again in this case y that is the marks scored would be the dependent variable, since it is dependent on the number of hours spent studying for the exam.
What does x i.e (x1) signify in the equation?
From the same equation mentioned above, x is the independent variable (IV), here in case of Simple Linear Regression, we have only one independent variable i.e x1.
This is the variable that is causing the dependent variable to change. From the example mentioned above the years of experience and number of hours spent studying are the independent variables.
What does b1 signify in the equation?
Here, b1 is the coefficient for independent variable i.e x1. This variable(b1) actually decides how a unit change in x1 influences y. Think of it as a multiplier or a connector that connects x and y.
and then finally comes b0, which is a constant which I will explain in detail in the later section of this article.
Understanding SLR with an Example:
The basic example of Salary vs Years of Experience where Experience (Years of Experience) is in the x-axis and salary is in the y-axis. Our main goal here is to understand how salary is dependent upon the years of experience.Here we have the data of different employees who are working in different companies.
This is how the Simple Linear Regression formula can be related to the above example:
The above formula can be read as Salary equals b zero plus b1 times experience. So what it essentially means is that it is putting a line through the above shown chart that best fits the data. I will explain about the best fitting line as we move ahead when I speak about Ordinary Least Square Method [OLS], but for now as you can see in the below mentioned picture the line that best fits the data.
Let us focus on the coefficients b1 and a constant b0.
The constant b0 is the point or value where the line intersects in the vertical axis i.e y-axis. Suppose let’s say b0 value is $30k, so when experience is 0, the second part of the equation i.e b1*experience becomes zero. That means salary = $30k. According to the model when a fresher joins a company his salary will be $30k.
Now, What is b1?
b1 is the slope of the line, more money you get as experience increases more will be the value of b1. As you can see in the above image when you perform the projections as per the black dotted lines, for one year increment in the experience there is a increase of around $10k in salary.
If the coefficient b1 is less, then slope will be less and even the salary increment per year will be less, if the slope is more then the experience will yield more increase in the salary and Yes, that’s how a Simple Linear Regression works.
How to find out the BEST FIT LINE FOR Simple Linear Regression [SLR]?
The answer is by Ordinary Least Square[OLS] Method
Now let’s try to understand how to find out the best fitting line or how SLR finds out that line for us.
The above shown graph is the same graph which I explained earlier. We have got the red dots that depicts the actual observation, we also have the straight line that best fits the data. To understand the working of OLS method let’s do some modifications on the graph:
We draw straight lines which are perpendicular to the observations to the best fitting line and then let’s select one observation as shown below:
Now you can see from the above picture that the red dot is the salary of a person for a particular year of experience. Let’s assume for 5 years of experience the salary is $50k. The model line, the blue line actually tells us what actually that person should get in terms of salary based on that data in generalized way. Let’s say he should earn $40K for 5 years of experience which is indicated by the green dot on the line.
Next, let’s call the red dot as yi that is the actual observation and green dot is called yi^(also called yi hat) which is the observation/value which the model is trying to predict and the blue dotted line is the difference between what the employee is actually earning and what he/she should be earning according to the model. In general, blue dotted line is the difference between the observed and the modeled.
To get this best fitting line, what is done is that we take the sum of (yi-yi^)², take the value of each one of those dotted blue lines, we square them and then wetake sum of those squares, once we have the sum of those squares we find out the minimum of them.
So, what a SLR does is that it draws lots and lots of these lines just like this:
and then finds a line which has minimum sum of squares of (yi-yi^) and that line is the best fitting line and the method followed to find out this line is called as the Ordinary least square [OLS] method.
I hope you found this article useful.
Thank you so much!
Feel free to connect with me either through LinkedIn, Instagram or Facebook.
I will be back with one more exciting article! Till then Stay Safe.
Cheers!
Arnold Sachith
