Overfitting and Human Behavior

Algorithms to Live By draws interesting parallels between familiar algorithms and human psychology. One of the more interesting chapters talks about overfitting in the context of human behavior.

Thinking as predicting

Our analytical thought process can be regarded as a form of supervised learning: Using your previous experiences and knowledge (the training data) to predict (build a model) some future outcome (the target).

How Overfitting ‘works’

• Every dataset that you will encounter in the real world contains some form of noise: essentially, patterns that are not pertinent to the task at hand.
• At the onset, a supervised-learner tries to reduce the error between its predictions and targets in the training data.
• This process is beneficial at first — the model keeps getting better and better at grasping useful trends.
• After a certain point however, the learner starts gathering spurious patterns that help with the training data, but don’t really generalize well. This, is called overfitting.

How we tend to overfit

Overfitting isn’t necessarily ‘overthinking’. It happens any time you read too much into your notions (whether right or wrong) to optimize something in your future.

The most relatable example is health & fitness: Humans have a ‘taste’ for fat and sugar, because these nutrients were very scarce in the prehistoric world. As a result, evolution gave us a sharp affinity towards their flavor. But now that we can dump copious amounts of sugar in everything we eat, Darwin’s laws haven’t been able to catch up. This is an instance of us biologically overfitting without even realizing it.

On the flip side, people also have a tendency to overdo things that they think will get them fit. We overtrain, buy into ineffective fads, face dietary issues…the list goes on. In each case, the tendency to (wrongly) extrapolate what we think will work shines through.

In a more extreme example, the book mentions training scars in military schooling: getting so used to preset situations in training, that they fail to do the right thing when confronted with a new circumstance in the field.

ML-inspired solutions

Algorithms to Live By mentions several ways to avoid overfitting, whether knowingly or otherwise. Let me draw analogies with ML terminology, so it makes more sense for data science folks:

Solution 1: Cross Validation

How? Use different metrics to track your progress.

Example: If body-weight has been your only indicator for effective weight loss, try measuring your body-fat percentage once in a while.

Solution 2: Better sampling

How? Add variety to your training and experiences.

Example: As part of your personal projects, try solving problems from domains other than the one you usually work with.

Solution 3: Regularization

How? Use heuristics (rules of thumb) when your confidence in interpretations is low.

Example: There’s a reason why index funds are considered a better investment-heuristic than actively investing (unless thats your day job).

Solution 4: Ensembling

How? Review diverse opinions.

Example: Before you buy the new-latest-awesome gadget your one colleague told you about, try looking up multiple reviews online (or from people you know).

Solution 5: Early Stopping

How? “Jump toward the bandwagon, not on it.”

Example: Instead of going all-out on a new fitness regime and then burning out, try sampling it in small ways first.