The Clustering Illusion

This post is the tenth part of a series of business-friendly posts, looking at cognitive biases and how they impact our ability to communicate and operate effectively.

What is it?

Humans have a tendency to assume streaks or patterns are non-random in small samples of what are really completely random distributions. We try to create order from chaos. This isn’t surprising really — the brain is a wonderful pattern-recognition machine and it serves us phenomenally well in this way…usually.

Why it occurs:

It comes down to this: humans have a tendency to under predict the amount of variability in small samples of random data. We then believe that this small sample size represents the larger population.

Where might you see it occur in real life?

The illusion is central to what is also known as the “hot hand fallacy”, which became famous in basketball but could easily apply to almost any sporting endeavour. Commentators would judge that a player shooting several successful free throws in a row was on a hot streak when in fact their success was at or close to the norm.

It isn’t just sporting culture that is impacted either. During WWII, it was commonly believed that bombs were targeted at particular areas as people looked for and saw patterns, but it was in fact a random distribution. People look for and see patterns in stock market fluctuations — but all they are seeing is a run of similar events in a row and what looks like a pattern in a very small sample size. Add that data into the data from the past five years and it will all even out into expected behaviour.

Why is it important?

When it starts to be particularly detrimental is when the person involved believes they have helped that success come about — it often leads to persisting with behaviour that was erroneously linked to successful outcomes. Imagine believing that your negotiation activity was the reason why several big business deals closed in a row when in fact other factors were in play and the timing was pretty much random.

How can this impact your business?

If performance (either your own, your team or the wider business) seems to be “hot”, question whether or not it really is, or whether you are looking at a sequence of events that is too small to make any real judgements about. If you’ve just closed three deals in quick succession, does that mean you are on a hot streak or is it just a random series of events that will even out over the course of the year? Be careful assuming the former as it might perpetuate poor behaviour.

If you are building an argument and need to demonstrate success in some way, try hard not to get caught up interpreting a cluster of data as a streak or pattern of success — it will be easily picked apart by someone who knows what they’re talking about.

And if you are receiving a pitch, be careful not to attribute too much importance to small data sets showing “great success” — it is just as likely to be “random success.”

If you want to know more please feel free to connect or check out a collection of cognitive biases here. Next up we will take a look at the Gambler’s Fallacy.