Overfitting and Underfitting

Before understanding overfitting and underfitting, let’s understand what is bias and variance.

Bias: Bias is like having a strong opinion or belief about something, and you stick to it no matter what. In machine learning, it’s about the assumptions your model makes about the data. A model with high bias might simplify the problem too much, making it unable to capture the true underlying patterns in the data. It’s like wearing blinders that prevent you from seeing the full picture. For example, if you’re trying to predict housing prices and your model always underestimates the prices, regardless of the features of the houses, it has a bias towards lower prices.

Variance: Variance is like being easily influenced by different situations. In machine learning, it’s about how much your model’s predictions change depending on the training data it’s exposed to. A model with high variance is like a chameleon, changing its colors too much based on the environment. It might perform really well on the training data, but when faced with new, unseen data, it struggles because it’s too sensitive to the noise or random fluctuations in the training data. For instance, if you train a model to recognize dogs and it gets confused by new pictures of dogs because it’s only seen a limited variety during training, it has high variance.

Overfitting

Low Bias and High Variance

Overfitting occurs when a model captures noise or random fluctuations in the training data as if they were significant patterns. This happens when the model is too complex relative to the amount of training data available. In essence, the model becomes highly tailored to the quirks of the training data and fails to generalize well to new, unseen data.

Key points:

1. High Model Complexity: Overfitting often arises from models with high complexity, such as those with a large number of parameters or a high degree of flexibility.
2. Memorization of Training Data: An overfitted model essentially memorizes the training data instead of learning the underlying relationships. This leads to poor performance when applied to new data.

Underfitting

High bias and High Variance

Underfitting occurs when a model is too simplistic to capture the underlying structure of the data. This typically arises when the model is not complex enough to represent the true relationships present in the data. As a result, the model performs poorly both on the training data and on new, unseen data.

Key points:

1. Low Model Complexity: Underfitting is characterized by models with low complexity, such as those with too few parameters or insufficient flexibility to capture the true data distribution.
2. Failure to Capture Patterns: An underfitted model fails to capture the essential patterns and relationships in the data, resulting in a poor fit to the training data and suboptimal performance on unseen data.
3. Insufficient Training: Underfitting can also occur when the model is not trained for a sufficient number of iterations or epochs, leading to incomplete learning of the data patterns.

Techniques to balance overfitting and underfitting.

• Model Regularization
• Cross-Validation
• Feature Engineering
• Ensemble Methods