The fragile “wisdom” of crowds

Why prediction markets aren’t quite as amazing as economists once thought

Published in

The Physics of Finance

6 min readMar 19, 2014

A couple of years ago, prediction markets were all the rage. They seemed to provide a mechanism for pooling the wisdom of countless people — by making them put their money where their thoughts were—to make accurate predictions of anything from sporting events to presidential elections. Notably, they seemed to out-perform experts, at least in many cases. Prominent economists such as Justin Wolfers were vocal supporters of new firms such as InTrade, which organized markets to generate all kinds of predictions.

InTrade still seems to exist (here’s their website), barely, after running into serious trouble following a lawsuit by the US Government. This recent article by Andrew Rice offers a nice history both of what happened to InTrade and the prediction market frenzy that, for now, seems to have died down a little. There is a lot of support for prediction markets from academic economists—see this short paper from Science a few years ago, for example. But, in general, this enthusiasm seems to rest on a fairly blithe assumption that the wisdom of crowds effect really is trustworthy — that crowds of people betting on things really should give very accurate results. Should it? Why should we believe that?

James Surowiecki, of course, wrote a bestselling book on the topic entitled The Wisdom of Crowds. But he was actually quite careful at the outset to acknowledge that the idea only works in some rather special situations (not that readers paid much attention). A crowd estimating the number of marbles in a jar or the correct price of a stock will only get superior results -- superior in accuracy to the guess of any one individual, and even of experts -- if the people are on average unbiased in their estimates; it won't work if they tend systematically to estimate too high or low. Moreover, the people have to make their estimates independently of one another. Any kind of social influence, one person copying or even being slightly swayed by the actions of another, also spoils the result. Wise crowds very quickly become dumb herds.

A fascinating experiment from a couple of years ago set out to test the wisdom of crowds effect in a controlled way, and the results should be more widely known. They show, in brief, that the wisdom of crowds effects is extremely fragile to an kind of social influence, i.e. the possibility for one person’s view on something to influence what other people about that thing. It depends entirely on people making totally INDEPENDENT judgements, and that is a hard thing to achieve, especially in today’s world of incessant Twitter and Facebook enabled gossip. In short, we should be very suspicious of any claims that prediction markets are amazing engines of wisdom.

OK, some detail. The experiments I’m thinking of were carried out by Jan Lorenz and colleagues from ETH-Zurich, and published in PNAS. Their idea was to use a crowd of 144 student volunteers and have them perform estimation experiments in a range of conditions. They gave the participants monetary incentives to estimate accurately, and chose questions (on things like geography and crime statistics) for which the true answers are known (like, how many murders were there in Zurich last year?). Then, in some trials, participants made their estimates on their own, without having any idea about the estimations of others, and in other trials, they were either informed in complete detail of what others had estimated or given at least average information on the others' estimates. The idea was to compare how well the crowd made estimates in the absence and presence of social influence.
What the results show is that social influence totally undermines the wisdom of crowds effect, and does so in three specific ways. It's interesting to consider these in some detail to see just how this whole "wise crowd" illusion falls apart in the face of a little social influence: 1. In what the researchers call the “social influence effect,” the mere act of listening to the judgements of others led to a marked decrease in the diversity of the participants estimates. That is, the estimates of the various people become more like one another -- people adjust their views to fit more closely with others -- but this does very little to improve the collective accuracy of the crowd. In effect, people think they are sharing information, but little information actually gets shared. The figure below illustrates what happens: in successive trials, a measure of the group's opinion diversity decreases dramatically if people hear either full or average information on the estimates of others, meanwhile the collective error decreases only marginally.

2. A second and even more interesting effect is what the researchers call the “range reduction effect.” Imagine that a government tries to use the wisdom of crowds, assembling a group and surveying their opinions, hoping to get a range of views and some idea of how much consensus there is on some topic. You would hope that, if the crowd's estimate was NOT accurate, this lack of accuracy would be reflected in a wide range of estimates from the individuals -- the wide range would signal a lack unanimity and confidence. A truly bad outcome would be a crowd that at once gives a very inaccurate estimate and does so with a narrow range of opinion differences, signalling apparent strong certainty in the result. But this is precisely what the research found -- in the social influence conditions, the individuals' estimates didn't "bracket" the true answer, with some being higher and others lower. Rather, the group narrowed the range of their views so strongly that the truth tended to reside outside of the group's range -- they were both inaccurate and apparently confident at the same time. 3. Finally, and worse still, is the “confidence effect”. The researchers interviewed the participants in the different conditions, asking them how confident they were in the accuracy of the group's final consensus estimate. Social influence, while it didn't make the crowd's estimate any more accurate, did fill the participants with strong confidence and belief in improved accuracy. Think 2005, housing bubble, mortgages with no income and no assets, etc. As hard as it is to imagine that people could have believed the market could not fail to go up further, most did. And they did in large part because they saw others apparently believing the same thing.
Altogether, this careful study points more toward the idiocy of crowds than their wisdom. Social influence is hard to eradicate. Even in markets, supposedly driven by anonymous individuals making their own estimates, lots of people are reading the newspapers and news feeds and listening to analysts, and, even when not, looking to price movements and using them to infer whether someone else may know something they don't. In these experiments, social influence makes everyone think and do much the same thing, makes it likely that the consensus view aims well wide of the actual truth, and, perversely, makes everyone involved increasingly confident that the group knows what it's doing. Some kind of Wisdom.

Read Andrew Price’s article about InTrade here.

Follow The Physics of Finance on Twitter: @Mark_Buchanan

Other recent articles:

High-frequency trading: algorithms now rule the world

Why the controversy over HFT misses the most important point

Why we misunderstand capitalism

Wealth isn’t money. It’s having solutions to human problems.

Taxis 2.0: Streamlining city transport with graph theory

Big city taxi systems could be 40% more efficient with device-enabled taxi sharing

Arrogant physicists — do they think economics is easy?

No. But their ideas can help improve economics, and here’s why

Random oblivious transfer: is quantum secured finance almost a reality?