What Does It Really Mean for an Algorithm to ‘Learn’?

Two general perspectives and some psychology

When one first encounters machine learning, one often rushes through algorithm after algorithm, technique after technique, equation after equation. But it is afterwards that one can reflect on the general trends across the knowledge that they have acquired.

What it means to ‘learn’ is a very abstract concept. The goal of this article is to provide two general interpretations of what it means for a machine to learn. These two interpretations are, as we will see, two sides of the same coin, and they are treated ubiqitously across machine learning.

Even if you are experienced in machine learning, you may gain something from temporarily stepping away from specific mechanics and considering the concept of learning at an abstract level.

There are broadly two key interpretations of learning in machine learning, which we will term loss-directed parameter update and manifold mapping. As we will see, they have substantive connections to psychology and philosophy of mind.

Loss-Directed Parameter Update

Some of the machine learning algorithms previously discussed adopt a tabula-rasa approach: they begin from a ‘blank slate’ random guess and…