Kleros and Mob Justice: Can the Wisdom of the Crowd Go Wrong?

Comparing Schelling Point Justice to Traditional Justice

Twelve Angry Men is a 1957 American courtroom drama film which tells the story of a jury as they deliberate the conviction or acquittal of a defendant.

In Kleros, we resolve disputes through crowdsourcing. Specifically, we randomly select a group of jurors that rule on the case. To incentivize the jurors to be honest, they are rewarded or penalized based on whether or not they are coherent with the vote of others.

The basic idea is that, if you don’t have any information on how the other jurors are going to vote, you think that they are more likely to be honest than not. You might reason that this is the case if only because honesty is a distinguished choice that the juror population is likely to gravitate to. Namely, we expect voting honestly to be a Schelling point for rational, self-interested jurors.

There are two basic situations where the jurors don’t reach the “right” decision:

• The natural Schelling point of the case (the decision that an infinitely large group of jurors that are not communicating among each other would arrive at) is the “right outcome,” but the chosen jurors nonetheless don’t arrive at that outcome.
• The natural Schelling point of the case is “wrong,” namely it would result in an unjust outcome. This might be due to systematic biases in the juror population.

Kleros has a number of built-in defenses against the first failure case. Principally, our appeal system makes it difficult for such deviations to persist. We will do a more detailed analysis of these defenses in other articles. Here we will focus on the second kind of failure case.

Biases in Justice Systems

In the film classic Twelve Angry Men, a single juror blocks what would otherwise be a unanimous decision of a twelve person jury in a case carrying the death penalty. With this extra time to reflect, the jury more carefully considers the evidence and concludes that it is not sufficient to convict.

In the film, each of the eleven jurors who initially vote guilty has his own motivations, among them biases against the defendant’s racial and economic background, as well as merely a simplistic analysis of the available evidence leading to confirmation bias.

Similar effects could apply to Kleros juries as well. In traditional courts, the primary defense against juries falling into simplistic reasoning is convincing arguments from lawyers during the proceedings. In Kleros, the parties can submit evidence, and it is not unreasonable to imagine that someday there will be an industry of crypto-lawyers that counsel parties on what evidence to provide in high-stakes Kleros cases.

Blinding Jurors Against Information Generating Biases

An initial defense against jurors falling into biases based on irrelevant information such as the demographics of the parties is that Kleros cases are “blinded” against such information. When you create a contract (a real-world contract, written in English or some other human language) that designates Kleros as the arbitrator in case of eventual dispute, you include a cryptographic commitment to that contract in an Ethereum smart contract.

For this, it would have been sufficient to include the hash of the contract text. However instead, the contract is split into clauses and a Merkle tree is formed out of the hashes of these clauses. Then the root of this tree is included in the Ethereum smart contract.

Hence, the clause(s) that are relevant to the dispute can be revealed to the jurors without showing them the rest of the contract. Then, if the full contract would make clear, for example, the national origins of the parties, this might be able to be nonetheless concealed from the jurors. Note this idea is similar to the double blind peer review used by many academic journals, where authors’ names and affiliations are not shown to reviewers to avoid biases.

The use of the Merkle Tree structure enables Kleros to only show to jurors the information from the contract that they need to know, to avoid unnecessary biases.

This is, of course, an imperfect defense. Often, the nature of the case will require that demographic information on the parties be revealed. In a dispute where a small business owner is unhappy with a website for her business designed by a freelancer and wants a refund, the jurors will have to be provided with the website as evidence, which will presumably reveal a lot of information about the small business.

Moreover, as both parties possess the full contract, either could publish it in its entirety if they thought that would give them a tactical advantage. Nonetheless, a party might not want to publish information on the other party that is clearly irrelevant to the case for fear of appearing too blatant and causing a backlash among the jurors.

Arthur Miller’s play The Crucible tells the story of the witchcraft trials in Salem, Massachussets, in 1692/1693. A different case of a justice system gone awry.

Is There Any Such Thing as a Unbiased Justice System?

It is true that there will sometimes be situations where jurors will gravitate towards Schelling points that are unjust. However, these kinds of biases are also present in other justice systems.

Studies show that there are important differences in judicial outcomes in traditional courts based on such demographic characteristics of the defendant as race, gender, and immigration status when controlling for type of crime committed. In some cases, sentencing has even been seen to be substantially more or less severe depending on whether the sentencing hearing is before or after the judge has had lunch.

Part of the reason Twelve Angry Men is such a compelling film is that the premise of a jury rushing to a hasty judgment is tragically believable, while the idea that a lone voice of reason can change the tide is something we want to believe in.

In Kleros, jurors are penalized if they provably communicate their votes with each other, as a defense against such communication altering the Schelling point. So admittedly, a lone Kleros juror will not be able to gradually convince the others of his or her point of view as Juror 8 does in Twelve Angry Men.

However, the film should not necessarily be taken as representative of the role of deliberation in the traditional justice system. Indeed, studies show that, for better or for worse it is, in fact, quite rare for jury deliberations to end in a result that had minority support on the jurors’ first vote, particularly when this support is below a critical threshold of around one fourth to one third of the jurors.

In fact, game theoretical work modeling juror utility in traditional settings has concluded that, due to strategic voting, juries requiring unanimity can counterintuitively result in higher conviction rates of innocent defendants than juries not requiring unanimity.

Further work in this vein shows that in the model of not requiring unanimity, at least for small juries, better outcomes are reached if jurors do not deliberate as this avoids issues where jurors free-ride on the efforts of others and do not rigorously examine the case for themselves. Note that non-deliberative juries are used in the Brazilian legal system and were used in ancient Athens.

Law and Order in Ancient Athens, by Adriaan Lanni. A good way to understand how this non-deliberative jury system worked.

Indeed, the degree to which the result of Twelve Angry Men is exceptional is highlighted to the viewers by a dialogue between the characters, where it is pointed out the uphill battle Juror 8 faces in convincing any of the other jurors to reconsider, and the high likelihood that in case of a retrial any other jury will fall into the same biases as the first and convict:

Juror 7: What do you think you’re gonna accomplish? You’re not gonna change anybody’s mind. So if you wanna be stubborn and hang this jury, go ahead. The kid’ll be tried again and found guilty sure as he’s born.

Juror 8: That’s probably right.

This discussion should not be interpreted as a slight against the traditional court systems. We at Kleros have a lot of respect for the legal system. However, providing just outcomes is a thorny problem. Kleros aims to provide affordable and fast dispute resolution, most notably for the kinds of small claims that it is not worth taking to the existing justice system. One can not reasonably expect Kleros to also be more rigorous that the expensive and slow traditional justice systems of countries with a well-established rule of law.

Ultimately, we see that when facing situations where a juror population gravitates toward an unjust decision, Kleros has many of the same challenges as existing justice systems.

To quote Atticus Finch from To Kill a Mockingbird:

“I’m no idealist to believe firmly in the integrity of our courts and in the jury system — that is no ideal to me, it is a living, working reality. Gentlemen, a court is no better than each man of you sitting before me on this jury. A court is only as sound as its jury, and a jury is only as sound as the men who make it up”.

At Kleros, we seek to expand this “living, working reality” to offer dispute resolution in cases that are not adequately addressed by the existing system.

Learn More

Join the community chat on Telegram.

Visit our website.

Follow us on Twitter.

Join our Slack for developer conversations.

Contribute on Github.