A simple guide to Bias-Variance Trade-off — Part 1
A quick refresh of the main concepts of bias-variance trade off for model building.
The above image showing the 4 scenarios of bias variance mix is very common when trying to explain bias-variance trade-off.
But what is bias and variance ?
The aim of a good model is to have low bias and low variance. If we can know which scenario our model suffers from then we can apply the appropriate solution to reach the low bias and low variance quadrant.
The explanation to these concepts are often highly technical but we shall only keep to the intuition here.
Bias quantifies the gap between the average model fit between the target variable and predictor variables and the true relationship between the target variable and predictor variables.
This is why missing important features from the model or considering a functionally simple model often results in a high bias model as the average model fit ends up being different from the true relationship. This is also known as Under-fitting.
Average model fit — If we drew a large number of training samples (let this number be denoted by S) from the population of all observations and obtained the model fit for each training sample, we would have S model fits and S fitted values for each observation in the population. The average across the S fitted values is known as the average model fit.
Variance quantifies the spread of the model fits around the average model fit. Hence, if the model fits vary a lot around the average model fit across the training samples this indicates we have a high variance model.
Complex models, especially with small samples often lead to high variance models. This is also known as Over-fitting.
Symptoms and Remedy
When building a machine learning model we would not be able to compute the bias and variance values to diagnose which quadrant we fall in. We can only detect this by analyzing our model training and test errors.
Let’s try to understand the following for each of the 4 scenarios when building a machine learning model :
1.What are the symptoms for detecting each scenario ?
2.What causes them and what’s the solution ?
Symptom
Cause and Remedy
Bias-Variance Trade-off Decision Routes
Between the first model we build and the final model we traverse different decision routes till we strike the right balance between bias and variance.
Here are the different bias-variance trade-off routes for going from Low bias-High variance to Low bias -Low variance :
Here are the different bias-variance trade-off routes for going from High bias-High variance to Low bias -Low variance :
Here are the different bias-variance trade-off routes for going from High bias-Low variance to Low bias -Low variance :
That’s all for Part-1. In the next part of this article we explore the relationships between training error, test error and model complexity, training sample size.
