Bias–Variance Trade-off

There are two sources of error, namely Bias and Variance, which acts as a hindrance for any algorithm to generalise.

BIAS — When we model in a very simple, novice way, for example just a single linear equation prediction for an actual complex model. Of course because of this Model becomes Under fit and miss out various important insights and relations between variables.

VARIANCE — On the other hand, when we become over concern for a simple given data and fit a model in a very complicated way, it results in Over fit. So each noise and outlier will be considered as valid data point and modelled accordingly.

When we want to fit any model we want :-

1.) Generalise for any new data point given to the model for prediction. (Not to Under fit)

2.) Make a complex yet simple mode.l (Not to Over fit)

BIAS : — Expectation [predicted function (x)] — Expectation [actual function(x)]

VARIANCE : — Expectation [predicted function(x)²] — Expectation [actual function(x)] ²

Expectation [(actual output — predicted output)²] = (BIAS(predicted output))² + VARIANCE(predicted output) + variance of points

Since actual function is deterministic.

Expectation [actual function(x)] = actual function(x)

E[f(x)] = f(x)

E[actual output] = E[actual function(x) — error] = actual function(x)

E[y] = E [f(x) — e] = f(x)

variance[e] = σ²