The Problem Of Overfitting And How To Resolve It

Explaining How To Fix The Curse Of Overfitting In Python

In this article, I want to explain one of the most important concepts of machine learning and data science which we encounter after we have trained our machine learning model. It is a must-know topic.

This article aims to explain the following topics:

1. What Is Overfitting In A Machine Learning Project?
2. How Can We Detect Overfitting?
3. How Do We Resolve Overfitting?

Photo by Isaac Smith on Unsplash

Introduction — What Is Overfitting?

Let’s set the foundation of the concept first.

Let’s assume you want to predict the future price movements of a stock.

You then decide to gather the historic daily prices of the stock for the last 10 days and plot the stock price on a scatter plot as shown below:

The chart above shows that the actual stock prices are some-what random.

To capture the stock price movements, you assess and gather data for the following 16 features which…