Simple Linear Regression — Parameter Estimates Explained

Linear Regression is a statistical modeling technique that is used to estimate the relationship between variables. For example, a linear regression model can be used to predict house prices based on variables such as number of rooms, age of the house, size of the house, and economy. This article provides an explanation of simple linear regression using an example to illustrate the steps in calculating the regression parameters. The same is then calculated using matrix algebra.

Simple Linear Regression

In simple linear regression, a relationship is established between two variables, an independent or predictor variable x and a dependent or response variable y. Lets create a regression model where we predict house prices based on a single variable, age of house (# of years since the house was built). Equation 1 defines this linear relationship between age (x) and price (y) of a house.

The statistical error accounts for error that results from the inaccuracy in modeling and measuring the relationship between x and y.

It is important to note that the parameters b0 and b1, and the error term ϵ are unknown. These are population parameters which are theoretical values and cannot be determined. We estimate these parameters from a sample of data collected on x and y.

We will explain these concepts using an example with a few (x, y) observations as shown in table 1.

Data from table 1 is plotted in figure 1. The scatter plot shows a negative relationship between age and price of house. For newer houses, prices will be higher than for older houses.

Estimating Regression Parameters

The most common method used to estimate the parameters b0 and b1 is the method of least squares. According to this method, the regression parameters are estimated by minimizing the sum of squared errors, the vertical distance of each observed response from the regression line. The least square method yields equations 2 and 3 which are used to find the estimated parameters b̂1 and b̂0 for b1 and b0, respectively.

Equation 5 is the regression line that is used to estimate y for given values of x. The regression line is plotted in figure 2. The line gives ŷ (pronounced y-hat), the predicted values of y, for different values of x. Some of the observed values of y are above the regression line and some are below. The difference (y — ŷ) is the prediction error called the residual. The regression line is the line of “best fit” as this line minimizes the sum of squared errors of prediction.

Table 3 shows the prediction error for each observation.

Any line with different values for the parameters b̂0 and b̂1 will give a sum of squared errors that will be larger than what is achieved from the line of best fit. Table 4 and Figure 3 illustrate this by using a different value for b̂0 and b̂1.

.

In figure 3, the regression line in blue is the line of best fit. The line in red is the line with b̂0 = 400 and b̂1 = -14. This line has a residual sum of squared errors of 89125 compared to 4418 for the line of best fit. You can use this link to download an Excel spreadsheet and try different values of b̂0 and b̂1 and observe the change in the regression line and the corresponding sum of squared errors.

Simple Linear Regression Using Matrix Algebra

Here, we will use matrix algebra to determine the line of best fit. A simple linear regression is expressed as:

Our objective is to estimate the coefficients b0 and b1 by using matrix algebra to minimize the residual sum of squared errors.

A set of n observations (x, y) can be written in the form:

Write the above equations in matrix form as:

y is a (n x 1) dimension vector that represents the prices of n houses.

X is a (n x 2) dimension matrix where each column represents the values for each predictor and each row represents the values of predictors for a house.

b is a (2, 1) dimension vector of parameters.

ϵ is a (n x 1) dimension vector of errors.

The linear regression model can now be written as:

y = Xb + ϵ

Estimating Regression Parameters Using Matrices

As explained in the previous section, we will use the method of least squares to estimate regression parameters. The least squares method finds the vector b by minimizing the sum of squared errors:

This gives a vector b̂ which is an estimate of b.

Using the observations on house age and price, the following illustrates the steps to calculate the parameters by using matrix algebra.

The linear regression model is:

where,

Therefore,

b̂0 = 396.92, and

b̂1 = -6.88

The regression equation is:

ŷ = 396.92–6.88x

Using this equation, the error values for each observation can be determined as shown previously.

There are several statistical software packages available which can provide the complete regression analysis including several statistical tests to determine the accuracy and significance of the model. Here, an attempt is made to illustrate the steps in calculating the parameters using equations as well as matrix algebra. Today, linear regression using least square estimates is widely being used in machine learning. Having a good understanding of the basic concepts of linear regression will be very helpful in performing advanced machine learning modeling.