Linear Regression With One Variable

Prediction with Gradient Descent and Normal Equation

Linear regression is a linear model, e.g. a model that assumes a linear relationship between the input variables (x) and the single output variable (y). More specifically, that y can be calculated from a linear combination of the input variables (x).

In this paper, we will fit the linear regression parameters θ to our dataset using gradient descent.

The dataset was taken from here

First, let’s import the libraries we need

We take a look at the data

We define the input variable (X) and the output variable (y)

Before starting on any task, it is often useful to understand the data by visualizing it. For this dataset, you can use a scatter plot to visualize the data, since it has only two properties to plot (profit and population).

Gradient Descent

In part, we will fit the linear regression parameters θ to our dataset using gradient descent.

Update Equations

The objective of linear regression is to minimize the cost function

where the hypothesis hθ(x) is given by the linear model

The parameters of your model are the θj values. These are the values you will adjust to minimize cost J(θ). One way to do this is to use the batch gradient descent algorithm. In batch gradient descent, each iteration performs the update.

Implementation

We have already set up the data for linear regression. In the following cell, we add another dimension to our data to accommodate the θ0 intercept term. Do NOT execute this cell more than once.

Computing the cost J(θ)

As you perform gradient descent to learn minimize the cost function J(θ), it is helpful to monitor the convergence by computing the cost. In this section, you will implement a function to calculate J(θ) so you can check the convergence of your gradient descent implementation.

Gradient descent

Next, you will implement gradient descent

After you are finished call the implemented, We initialize the θ parameters to 0, iteration to 2000, and the learning rate α to 0.5.

Our final value θ, we will use to predict weight for height 1.52

Normal Equation

The closed-form solution to linear regression is

Using this formula, you will get an exact solution in one calculation: there is no “loop until convergence” like in gradient descent.

We calculate theta

Our final value θ, we will use to predict weight for height 1.52

Comparing With Sklearn Liniear Regression

Now we compare it with linear regression using sklearn

We first determine the features and targets

then we use sklearn for linear regression

Our intercept and coefficient, we will use to predict weight for height 1.52

From the results above, we can see that the calculation results between the normal equation and linear regression sklearn have the same results for the intercept and the coefficient. However, calculations using gradient descent give different results.

To overcome this, it is necessary to set the iteration and learning rate α to a different value so that θ values can be obtained that is close to the calculation with the normal equation and sklearn linear regression.

Thank You