Linear Regression — A Simple Statistical Regression Method
Linear Regression is a statistical method that is used for predictive analysis and in this article, we will talk about linear regression, how does it work, and how it can be used in several machine learning problems.
Linear Regression is an algorithm that performs regression and is based on supervised learning. If you need to solve a problem like forecasting, prediction or you want to develop a relationship between variables then linear regression is the right algorithm that can be used as a solution. Before talking about linear regression, let’s understand what actually regression is.
Regression
Regression models are used for the prediction of continuous values and fall under the domain of supervised machine learning. The main purpose of the regression model is to determine a relationship between variables by estimating how one variable affects the other. A simple example of regression is that it can help predict sales for a company based on previous sales and GDP growth etc.
Now as we know what regression is let’s talk about the details of linear regression. Simple linear regression is a model in which there is only one independent variable and it attempts to create a relationship between an independent variable(x) and a dependent variable (y) but in higher dimensions, we can have more than one input variable.
The above image depicts an example of linear regression where the red line is considered to be the best fit straight line. Provided the data, we try to plot the line which fits best at the data.
As we have talked about the independent variable(x) and dependent variable(y) then the following equation can be used to model the line to the given data:
Linear Equation
Y = a + bX
In the above equation, X is the independent variable and Y is the dependent variable, a and b are coefficients whereas b is the slope of the line and a is the intercept.
As we have talked above that we try to plot the line which fits best at the data, which means that we will be wanting the best values for a and b. For finding the best values of a and b there are several methods. The most common out of them is Least-Squares Regression, so let’s talk about it.
Least-Squares Regression
In the least-squares regression technique, we try to minimize the sum of the squares of the vertical deviations from each data point to the line. It means that if a point lies on the line exactly, then its vertical deviation is 0. In least-squares regression, the deviations are squared and then summed up.
At this point, you must be thinking that we have calculated the error but how do we minimize it or how do we choose the line that best fits the data so our error is minimal. At this point, a concept which is known as Gradient Descent comes into play. I won’t be going into the mathematical steps of Gradient Descent but would like to give you an overview of how gradient descent works.
Gradient Descent
By looking at the above image, you can see that how after some iterations gradient descent finds the best fit line to the data. Gradient Descent is an algorithm that updates the values of a and b to minimize the error in the deviations. The idea of gradient descent is to start with some values of a and b and then these values are iteratively updated so that we can get the minimal error and our line best fits the provided data. In order to get the best values of a and b, we need to take some steps and these steps are known as the learning rate. So what should be the learning rate kept, in order to find the best values? The following image denotes what amount of learning rate can be kept in order to find the best values in an optimized way.
For example, if we keep a big learning rate then we can reach sooner to our best values but there is a chance that we might overshoot the best values and we find other values that are not the right fit. If we keep a very small learning rate, we would find the best values but it might take a very long time to find those values. So the learning rate should be kept right and this will decide on how fast the algorithm converges to minima.
Code
Following is a simple implementation of linear regression. If you need to understand the code in-depth, I have provided a link in the references section of this article that refers to this code.
import numpy as np
import matplotlib.pyplot as pltdef estimate_coef(x, y):
# number of observations/points
n = np.size(x)# mean of x and y vector
m_x = np.mean(x)
m_y = np.mean(y)# calculating cross-deviation and deviation about x
SS_xy = np.sum(y*x) - n*m_y*m_x
SS_xx = np.sum(x*x) - n*m_x*m_x# calculating regression coefficients
b_1 = SS_xy / SS_xx
b_0 = m_y - b_1*m_xreturn (b_0, b_1)def plot_regression_line(x, y, b):
# plotting the actual points as scatter plot
plt.scatter(x, y, color = "m",
marker = "o", s = 30)# predicted response vector
y_pred = b[0] + b[1]*x# plotting the regression line
plt.plot(x, y_pred, color = "g")# putting labels
plt.xlabel('x')
plt.ylabel('y')# function to show plot
plt.show()def main():
# observations / data
x = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
y = np.array([1, 3, 2, 5, 7, 8, 8, 9, 10, 12])# estimating coefficients
b = estimate_coef(x, y)
print("Estimated coefficients:\na = {} \
\nb = {}".format(b[0], b[1]))# plotting regression line
plot_regression_line(x, y, b)if __name__ == "__main__":
main()
Output
Estimated coefficients:
a = 1.2363636363636363
b = 1.1696969696969697
Conclusion
Linear Regression is an algorithm that can be a great way of solving problems for Machine Learning Enthusiasts. It can be helpful while evaluating the trends and make forecasts or can be used to assess risk in financial services or can be used to make predictions for continuous numerical values.
That’s all for now! I hope this article has helped you in understanding Linear Regression. Feel free to put a comment below if you have any suggestions or questions.
