Understanding Bias and Variance in Machine Learning
The terms bias and variance describe how well the model fits the actual unknown data distribution. In general one never has a dataset that fully replicates the true data distribution. Instead one usually only has a small subsample. A good model is able to approximate the true but unknown data distribution, based on the given training data.
Bias
Bias is the difference between the average model prediction and the actual data. Hence, high bias means that the decision boundary leads to a high error rate (or loss). Thus, it fails to capture complex relationships and indicates the model might be to simple. In contrast, low bias means that the model has a good fit with the training and test data.
Variance
Variance describes if a models predictions are sensitive to changes in the training data. In particular variance describes the error when using different batches of training data. Figure 2 illustrates the term variance.
Imagine, we only have training data 1 as our dataset. We train a model and the middle part of the illustration shows that our trained decision boundary perfectly fits the training data. However, if we would have a different sample of the true but unknown data distribution, for example only data points from training data 2, then we see…
