Data Mining Reveals How The “Down-Vote” Leads To A Vicious Circle Of Negative Feedback

A classic theory of behavioural psychology predicts that punishment should improve behaviour. But the first study of online voting behaviour in social networks shows exactly the opposite

In the 1930s, the American psychologist Burrhus Skinner popularised the notion of operant conditioning, the notion that an individual’s future behaviour is determined by the punishments and rewards he or she has received in the past. It means that specific patterns of behaviour can be induced by punishing unwanted actions while rewarding those that are desired. And it certainly works with rats and pigeons.

This idea has since become one of the foundations of behavioural psychology and is an important driver of the way online social networks are designed and operate. Many have systems that allow people to like, or up-vote, certain types of content while disliking, or down-voting, others. An up-vote can be thought of as a reward designed to encourage while the down-vote is a punishment designed to discourage.

In theory, this should guide contributors towards producing better content that is more likely to be rewarded. At least, that’s what the theory of operant conditioning predicts.

But does that actually happen on real social networks? Today, we find out thanks to the work of Justin Cheng at Stanford University in Palo Alto and a couple of buddies.

These guys have measured how up-voting and down-voting influences the behaviour of a large number of contributors to different social networks. And they say that the results are far from reassuring.

The evidence is that a contributor who is down-voted produces lower quality content in future that is valued even less by others on the network. What’s more, people are more likely to down-vote others after they have been down voted themselves. The result is a vicious spiral of increasingly negative behaviour that is exactly the opposite of the intended effect.

Cheng and co began by compiling a dataset of the comments associated with news articles on four online communities: CNN.com, a general news site; Breitbart.com, a political new site; IGN.com, a computer gaming news site; and Allkpop.com, a Korean entertainment site. The data includes 1.2 million threads with 42 million comments and 114 million votes from 1.8 million different users.

These guys conducted a survey on Amazon’s Mechanical Turk asking people to rate the quality of comments from these communities and then worked out the how the percentage of up-votes the post received correlated with the human evaluation. This confirmed that the percentage of up-votes is indeed a good measure of the quality of a post.

Cheng and co then built a machine learning algorithm that predicts a post’s quality by examining the words it contains. They trained the algorithm on half of the posts in the community and found that its ratings correlated well with the subjective opinions of humans. So the algorithm is an automatic way of rating the quality of every post in their dataset.

Then came the actual experiment. They used the machine learning algorithm to find posts of equal quality but with a twist: they matched posts into pairs in which one had been positively received by the community while the other had been negatively received. In other words, one of these posts received more up votes while the other received more down votes.

They then assessed the future output of the authors of these posts to measure the effect of positive and negative evaluations.

The results are something of an eye-opener. “We find that negative feedback leads to significant behavioural changes that are detrimental to the community,” say Cheng and co.

“Not only do authors of negatively-evaluated content contribute more, but also their future posts are of lower quality, and are perceived by the community as such,” they say. And it gets worse: “These authors are more likely to subsequently evaluate their fellow users negatively, percolating these effects through the community.”

By contrast, positive feedback does not appear to influence authors much at all. It does not encourage them to write more and does not improve the quality of their posts. Curiously, authors that receive no feedback, are more likely to leave the community entirely. “Surprisingly, our findings are in a sense exactly the opposite than what we would expect under the operant conditioning framework,” say Cheng and co.

That points to an obvious strategy for improving the quality of comments on any social network site. Clearly, providing negative feedback to “bad” users does not appear to be a good way of preventing undesired behaviour.

So how can unwanted behaviour be stopped? “Given that users who receive no feedback post less frequently, a potentially effective strategy could be to ignore undesired behaviour and provide no feedback at all,” say Cheng and co.

That’s an interesting study that provides a fascinating insight into the complex nature of social interactions. And it backs up certain kinds of real life experience. Anyone with children will know that it is possible to unintentionally reward bad behaviour with the increased attention that it generates. So sometimes it’s better to ignore unwanted behaviour than to focus on it.

But at the same time, parents also need an effective way of stepping in and actively preventing more serious incidents of undesirable behaviour when necessary.

The work of Cheng and co clearly suggested ignoring bad behaviour is an effective way of discouraging it, and one that social network sites might profitably explore. But at the same time, these sites will need a way to step in and actively prevent certain types of behaviour when necessary.

What’s needed now, of course, is a test of this idea. There are certainly social networks that allow up voting but not down voting (Medium being one of them).

An interesting question is whether it results in the same rich tapestry of opinion that clearly flourishes on social network sites that allow both types of voting. In other words, does this kind of manipulation have other consequences that Cheng and co have not yet accounted for.

Clearly, there’s interesting work ahead in teasing these things apart.

Ref: arxiv.org/abs/1405.1429 : How Community Feedback Shapes User Behavior

Follow the Physics arXiv Blog on Twitter at @arxivblog, on Facebook and by hitting the Follow button below