Stop Using Data Like a Drunk Uses a Lamppost
Quantitative info is for illumination, not support. To succeed in the future, companies must account for uncertainty and learn not to trust lazy predictions.
Market Research Is Important
Let’s start with a given: market research is important.
Doing that research is an attempt to mitigate the uncertainty of the future of any work you may be doing.
It helps you understand the nature and availability of the people you’re trying to sell to. Launching a startup? Building a product? Creating an ad campaign? Do your market research.
These days, market research is loosely divided into two broad methodologies: quantitative and qualitative.
Qualitative: having in-depth discussions with a few people, alone or in groups, to get more substantive understanding.
Quantitative: asking large numbers of people questions and mathematically weighting and modeling the results into something statistically representative, commonly through polls or surveys.
Polls are a Favored Way to Do Research
Ah, polls.
Everybody loves the idea of polls.
That’s because polls make the complex seem simple. They take a thoroughly complicated system—i.e., human thoughts and feelings—and reduce those human thoughts and feeling to data.
But here’s the problem: Polls are not, and have never been, predictive. There are many reasons for this, but most especially because humans are terrible predictors of their own behavior.
Of late, the primary mechanism for mass polling has been calling thousands of random phone numbers. That mechanism has become less efficient because the vast majority of people are no longer willing to answer the phone or the questions. Response rates are so low that it’s becoming hard to appropriately weight the data.
We only ever get answers from people who like to answer the phone and chat to pollsters or ones that fill out surveys online — and how often do you do that?
These days, polls seem to tell us the opposite of what’s going to happen, while also providing dangerous signals to the market beforehand.
Exhibit A: Brexit.
Exhibit B: Trump.
Some academics are insisting that “polls never will be right again” and should therefore not be reported in the media.
But Polls Have Always Been Wrong
Despite rigorous pre-testing, the vast majority of new products brought to market fail.
And despite researchers concluding that “no other product has ever performed so poorly” in taste tests, Red Bull is one of the most valuable brands in the world.
So where does that leave us? How then, when we invent and launch new things, can we gain insight and give ourselves the best chance of success?
The facetious answer: just run polls and assume the opposite of what they say! But let’s try to be more constructive.
The standard answer is probably something about the intersection of social media and big data. And if we look, we find that there are indeed interesting developments in this intersection of areas.
The company Remesh, for example, says it is “Replacing Polls and Surveys with Conversation”. They aggregate social media data and use it to build bots you can interact with that are representative of large groups of people.
The Failure Isn’t in Polls—It’s in Us
What Remish is doing is interesting, sure.
But the larger point is simply that we desperately hunger for simple, predictive answers that explain a complex, unpredictable world. This is a failure of understanding, not of research.
Researchers have always known and insisted that they are not prognosticators. But their research is often presented in a (necessarily) reductive manner. People then latch on to whichever points they wanted to be true in the first place. In other words, they use research like a drunk uses a lamppost—for support rather than illumination.
Taking a number of readings with whichever quant and qual tools and methods become available in the future, and triangulating as best we might, with all the objectivity we can muster, will still never produce certainty. Not when it comes to human behavior.
Aggregated human behavior in the real world is inherently unpredictable because it’s a complex system. Uncertainty is inherent to complex systems because of emergence — the same reasons the market and the weather can never be absolutely predicted. The interactions of innumerable elements create emergent effects that cannot, ever, be predicted — they are stochastic by their very nature.
So the failure doesn’t simply lie in the research, but in how we think about it. We must learn to never accept research if it attempts to peddle lazy predictions. To Nate Silver’s credit with regard to Trump, he continually insisted the models suggested a range of outcomes. People just don’t listen.
So research for the future of work isn’t just about finding new better techniques and technologies — it’s about understanding that unpredictability is built into the system and accounting for it.
Even when we would prefer it to be otherwise.
