ML World: Learn Machine Learning! Part 2

This tutorial answers all the following questions:

1. What is Regression Analysis?
2. What are the different types of Regression Analysis?
3. What is Simple Linear Regression?
4. How does Simple Linear Regression find the best fit line?
5. What are the assumptions of Simple Linear Regression?
6. How to Implement a Simple Linear Regression model?
7. When should we use Simple Linear Regression?
8. What is Cost Function?
9. What is Gradient Descent?
10. How is error calculated?

Now that we have completed our Data Preprocessing Series, it is time to dive right into Machine Learning and start building our models.

If you have not checked our previous series, I highly recommend you do so to get a good grip on some concepts we will require here as well.

This is it. This is the moment, This is the beginning of you becoming a Machine Learning Practitioner!

The simplest model of Machine Learning is Regression analysis.

Let us get started and get to know more about Regression.

What is Regression Analysis?

Regression analysis is implemented when you want to predict a continuous dependent variable (Target) from several independent variables (Features).

In simple words, regression analysis is predicting something.

What are the different types of Regression Analysis?

Different types of Regression Analysis methods are as follows:

1. Simple Linear Regression
2. Multiple Linear Regression
3. Polynomial Regression
4. Support Vector Regression
5. Decision Tree Regression
6. Random Forest regression
7. Logistic Regression

Woah! That is a lot to deal with at the moment. Let us get started with the simplest Machine Learning model: Simple Linear Regression.

What is Simple Linear Regression?

Simple Linear Regression/ Univariate Regression in ML is a Supervised Learning algorithm.

Simple Linear Regression allows us to summarize and observe the relationship between two continuous quantitative variables x and y, which is represented by a straight line.

In simple words, Simple Linear Regression is the manifestation of the equation of straight line having coordinates(x,y) given as:

In the above equation,

x: x is the independent variable or feature. It is the input.

y: y is a dependent variable or target variable which our model predicted/estimated.

m: m is the slope. It determines the angle at all times.

c: c is an intercept. It is constant that decides the value of x when y=0.

Similarly, The equation of Simple Linear Regression is as follows:

• y is the predicted value of the dependent variable/target variable (y) for any given value of the independent variable/feature (x).
• B0 is the intercept, the predicted value of y when the x=0.
• B1 is the regression coefficient: the amount of change in y as x tends to increase
• x is the independent variable/feature ( the variable we expect is influencing y or the controlling variable).
• e is the error function or the difference in the actual and predicted values.

Simple Linear regression finds the line of the best fit line through your data by searching for the regression coefficient (B1) that minimizes the total error (e) of the model.

How does Simple Linear Regression find the best fit line?

To understand this, let us consider an example.

Suppose we have a data set of grades for some students and need to predict the grades of students based on the time they spent studying.

Now, in this case :

y = Grades of students.

x = Time spent Studying.

Let us consider a trend line which represents the possible best fit line. In the above dataset, yellow stars represent the grades of students dependent on the time they spent studying.

Now let us draw a vertical line from these data points onto the best fit line. Thus, you can see where the dataset point is and where it should be placed according to the best fit line.

The yellow star indicates the actual value whereas the green dot represents the value observed by the Simple Linear Regression model.

Now, to get this best fitting line, you take this difference between the actual and estimates (y-y’) and square it and calculate its sum. Once you have the sum, you need to find the minimum out of them.

What a simple linear regression does is it draws lots and lots and lots of these lines and counts the sum of the squares every single time and records it and then it finds the minimum sum of squares.

And that line is called the best fit line, and this method is known as the ordinary least squares method.

What are the Assumptions of Simple Linear Regression?

For a simple linear regression model in practice, the model should conform to the assumptions of linear regression.

There are four assumptions associated with a linear regression model:

1. Linearity: The first assumption of simple linear regression is that the two variables (target and feature) should have a Linear relationship.
2. Homoscedasticity: The variance of residual is the same for any value of x.
3. Independence: Observations are independent of each other.
4. Normality: For any fixed value of x, y is normally distributed. In other words, it suggests that the linear combination of the random variables should have a normal distribution.

How to Implement a Simple Linear Regression model?

To implement the simplest ML model, first, you need to process the data. I suggest you check out our Data processing series to familiarize yourself with some concepts.

To start building a Simple Linear Regression Model, we first need a dataset and a problem statement. Let us consider a simple dataset having a dependent variable ‘Salary’ and independent variable ‘Years of experience’.

The datasets in real-world applications are endless, but to keep things simple and wrap our heads around the concept, we should start with a much smaller dataset.

Here we have a dataset of a few employees’ salaries corresponding to their years of experience.

The first thing we need to do is import all the necessary libraries. Here we need to import the LinearRegression class from sklearn.linear_model library.

Once we are through with importing libraries, we need to import our dataset.

After successfully importing the dataset and separating it into features and targets, we need to split the dataset into train and test sets.

Now, we need to create an object of the class LinearRegression() and train the model on the training set. After our model is trained, we need to test it on a test set.

We can then visualize the performance of our model on the training set and testing set to get a better intuition of its efficiency.

When to use Simple Linear Regression?

Simple Linear regression helps you in understanding and quantifying the relationship between your numerical data.

You can use simple linear regression when you want to know:

1. How strong the relationship is between two variables (for example the relationship between area and house prices).
2. The value of the dependent variable at a certain value of the independent variable (For example the price of a house when the area is given).

Learning Parameter for Simple Linear Regression: Cost Function

The goal of any ML model is to learn from data. To make predictions and learn from experience. It is similar to the analogy of a toddler learning life lessons and developing himself according to his/her experience,

Similarly, the cost function helps the learner to learn and correct his mistakes and develop from it to avoid future inaccuracies.

In ML, cost functions are used to estimate how badly models are performing.

A Cost function is a measure of how inaccurately the model performs in terms of its ability to predict the relationship between X and y.

This is generally expressed as a mean squared error, which measures the difference between the actual value (from the dataset) and the estimated value (the prediction). It looks like this:

The objective of an ML model, therefore, is to find these parameters, weights, or a structure that minimizes the cost function.

Function to minimize Cost Function: Gradient Descent

Now that you know that our model functions accurately once the cost function is minimized, we need to think of some parameters to do this task. Thus Gradient Descent comes into play.

Gradient descent is an efficient optimization algorithm that attempts to find a local or global minimum of a function.

What are Local and Global Minima?

Local Minima :

Functions can have an elliptical, hill or valley like structure: a place from where they reach a minimum or maximum value.

It may not be the minimum or maximum for the whole function, but locally it is.

Global Minima :

The minimum over the entire function is called an “Absolute” or “Global” minimum.

There is only one global minimum, but there can be more than one local minimum.

Well, I hope these definitions gave you a little insight. Now we can try and understand the concept of gradient descent much more easily.

Gradient descent enables a model to learn the direction that the model should take to reduce the difference between actual and predicted values.

Gradient Descent in the simple linear regression refers to how the model parameters b0 and b1 should be adjusted or corrected to further minimize the cost function.

As the model iterates, it gradually converges towards a minimum where further adjustments made in the parameters produce little or zero changes in the loss.

This point is also known as convergence. At this point the weights of the model are optimum and the cost function is minimized.

This iteration depends on the learning rate of the model (α).

The learning rate determines how big the step would be on each iteration.

• If α is very small, it would take a long time to converge and become computationally expensive.
• If α is large, it may fail to converge and surpass the minimum.

What are the limitations of simple linear regression?

Some limitations of simple linear regression are as follows:

1. So far, we’ve only been able to examine the relationship between two variables. Simple linear regression allows us to predict a target variable only when it is controlled by one feature.
2. It oversimplifies a real-world problem by assuming a linear relationship between variables.
3. Simple Linear Regression is sensitive to outliers. Outliers are the data points that lie outside the scope of the regression line in this case. Thus, accuracy is harmed in the presence of outliers.

Congratulations! I hope you were able to compute your first ever ML model: Linear Regression successfully. There are many concepts to wrap your head around, but with a little dedication, you will surpass this phase.

Keep practicing peeps!

(Image Source: Internet)