Everything you need to know about : Inductive bias

Constraints on the model!

Photo by Christian Lue on Unsplash

Learning is the process of looking for an explanation of observed data in a certain space of solutions. The best possible solutions need not be unique, so we can have multiple solutions to the same problem.

How do we choose a unique solution then?

Enter Inductive Bias!

Inductive biases allow the learning solution to prioritise one solution over another, independent of the observed data.

Why is inductive bias important?

1. By constraining the models, it reduces the space of solutions without sacrificing key metrics thus improving performance.
2. It allows us to encode our understanding of the world to search for viable solutions. This lets us steers the model generalization the way we want.

Let’s consider a few examples —

1. Linear regression : The inductive bias is on the data-generating process i.e this assumes that the data is generated via a linear process with additive gaussian noise. You can see that this is independent of the observed data as it is encoded in the model itself.