Does Big Data Reinforce Our Recency Bias?

I’m sure you’ve heard the stats. 90% of the world’s data was created in the last two years. Iphones are millions of times more powerful than the computer used to put a man on the moon. We upload 72 hours of youtube video and share 2.5 million bits of Facebook content every minute. It’s remarkable isn’t it?

However you look at it, the amount of data we produce store and analyse has exploded in the last few years.

In an article earlier this week, the BBC postulated that:

“Every two years for about the last three decades the amount of data in the world has increased by about 10 times — a rate that puts even Moore’s law of doubling processor power to shame.”

For data analysis this is great, we’ve now got exponentially more data to analyse and manipulate than we have ever had at any point in human history.

Is New Better?

The issue with analysing all this ‘new data’ lies in the name. It’s all new. Data analysis using big data systems is going to be vastly skewed by recency, because that is when the vast majority of data has been captured.

Investors talk about the concept of the ‘recency effect’ as a cognitive bias that undermines good decision making. Recency bias involves putting an increased emphasis on events that have happened in the immediate past and assuming that these patterns will have a direct correlation with what is about to occur in the immediate future.

While normally related to stock performance, understanding the recency bias is of vital importance to those looking leverage cognitive procurement solutions. While increased information about your suppliers or the market that you operate in may indeed lead to smarter more informed business decisions. You must realise that you are potentially analysing a statistically insignificant period of time. The performance of your market or supplier in the past six months or year may have little indication on their ongoing performance.

Reading too deeply into the data your newly established cognitive procurement program has spat out, could mean too great a focus on recent performance and too little consideration longer term performance. This may, in turn, result in rash decisions based on short term market movements or fluctuations.

While it’s true that history repeats itself, recent performance is no direct indication of future success or failure. As such, it’s important not to give this information a disproportionate amount of attention.

The challenge for procurement professionals will be to understand this vast amount of new data in relation to the relatively limited levels information we can glean from the more distant past.

Perhaps as big data solutions continue to develop a level of intelligence can be programmed in so that these systems are able work around and ignore irrelevant but recent information.