Understanding Bias Variance Trade-off for Machine Learning models: Part 1

Assumptions made by a model to map Input and Output

Hello World! My Blog for the Data Science Community. In this blog, we are going to discuss the Bias Variance Trade-off for the Machine Learning Models. If you trying to understanding the Bias Variance Trade-off then this blog is for you.

Sometimes it is challenging for practitioners or even entry-level professionals to understand what this term represents. Bias Variance Trade-off plays a major role in the selection of Machine Learning models. It’s necessary to understand Bias and Variance concept for every algorithm to create a better model.

Before discussing Bias-Variance Tradeoff, we have to know the basic concept of this. Let’s start with Basic

What is Bias Error?

People are often confusing Bias with residual and they say that its a prediction error, But Actually wrong. Even I got confused with these terms. To make it simple,

Bias is the error obtained with irreducible noisy error from the training dataset when the model tries to map the relation between independent and dependent variables.

Noisy error is an irreducible error because the model needs something more apart from the given information to map the relation, even if we try with any powerful models without sufficient data, and if data is inaccurate then, it will not get reduced.

Here the Feature Creation plays a major role in reducing the bias error, which obtained from the training dataset. Along with creating sensible features, we can also try to implement noise handling techniques such as using penalized models and performing some resampling techniques to reduce Bias error.

• Low Bias: Predicting less assumption about Target Function
• High Bias: Predicting more assumption about Target Function

What is Variance Error?

Variance is another important key terminology to understand and it will help to interpret the model performance in the estimation of the target function.

Variance measures how jumpy our estimator is, for example, the green curve in the previous graph is considered a model/estimator with high variance as it is very jumpy to fit the training data.

In the training process, the model tries to improve and accurately capture the regularities in the training data. By overdoing this process, the model ends up learning extra noise, which will make your model flexible enough to overfit on the training data. We can see an example of this in our classification example above — the green curve shows this in action, which will un-generalize your model.

• Low Variance: Predicting small changes to the estimate of the target function with changes to the training dataset.
• High Variance: Predicting large changes to the estimate of the target function with changes to the training dataset.

Why is Bias-Variance Tradeoff?

If our model is too simple and has very few parameters then it may have high bias and low variance. On the other hand, if our model has a large number of parameters then it’s going to have high variance and low bias. So we need to find the right/good balance without overfitting and underfitting the data.

This tradeoff in complexity is why there is a tradeoff between bias and variance. An algorithm can’t be more complex and less complex at the same time.

Forever there will be a relationship between bias and variance in machine learning.

• Increasing the bias will decrease the variance.
• Increasing the variance will decrease bias.

Bias-Variance Tradeoff Relation With Overfitting And Underfitting?

Till now we have gained some basic knowledge about Bias and Variance, now let’s move to some other important terminology used in Bias and Variance that are Underfitting and Overfitting.

• A model with low variance and low bias is the ideal model where every professional are expecting to build.
• A model with low bias and high variance is a model with overfitting.
• A model with high bias and low variance is usually an underfitting model.
• A model with high bias and high variance is the worst-case scenario, as it is a model that produces the greatest possible prediction error.

The mathematical equation that explains this relationship is:

Bias-variance tradeoff formula. From James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112). New York: springer.

The expression on the left is the mean square of the errors in the model predictions (MSE).

With the expression on the right, we can see that this error is explained by the square of the model bias, the variance of the model, and the variance of the inevitable errors.

To understand Overfitting and Underfitting in detail refer to my BLOG

Conclusion

In this blog, we have discussed in detail about Bias Variance Trade-off and we have also covered how it relates to Overfitting and Underfitting. In the next blog, we will discuss the Bias Variance Trade-off for Machine Learning models.

I hope this helps as a starter for exploring other concepts in Bias Variance Trade-off.

Also, let me know if I have missed out on anything in Bias Variance Trade-off.

If you liked the article, feel free to give me claps and help others to find it.

Connect with me in:- Linkedin

Connect with me in:- Github