By Charles Adjovu; Updated 2020–12–25

Peer Prediction Mechanism: Eliciting truthful information without verification

Podcast on the Peer Prediction Mechanism

What is the Peer Prediction (PP) Mechanism?

The PP mechanism is a mechanism designed to elicit truthful (i.e., honest) responses from a set of respondents when there reports cannot be verified because: 1) there is no objective ground truth available (i.e., inherently subjective circumstances), or 2) it is too costly to acquire the ground truth.

The PP mechanism relies on the correlation of a respondent’s response with the responses of other respondents to determine how to score their response. If their correlation is high (or rather, very predictive of other user’s reports), then the respondent will receive a payout. If the correlation is low, then the respondent will not receive a payout.

A couple of areas where we can see these problems is in online review sites such as Yelp and Amazon where users review and rank products and restaurants based on their experiences. On both Yelp and Amazon, we have many people giving reviews but we do not know the authenticity o these reviews, and more often than not, users are only giving 1’s or 5’s to the product or restaurant they are reviewing.

Now, can we really say how the user experienced the product or restaurant? Not necessarily. This is an inherently subjective area because the only person who can determine a user’s perception of their experience is the user, and to determine if every user was responded honestly, our costs would increase tremendously (we would have to determine if the user is real and how to find them, if they really went to the restaurant or bought the product, and find hard evidence to support their review).

Then back to the ratings, it is highly unlikely that a product or restaurant can only be rated as absolutely terrible with 1 (the lowest) and absolutely great with a 5 (the highest) so frequently. Why are users not giving 2’s, 3’s, and 4's?

Of course, this problem can also be explained by the fact that only those who have a very good or bad experience with a product or restaurant are the most likely to report their experience. However, was that truly their experience with the restaurant or product? Is it not possible that someone rated a product or restaurant a 1 to intentionally drive away business or a 5 to intentionally drive business towards the restaurant or product?

So if we want more accurate ratings from our users, this is where the PP mechanism can come in.

Issues the PP mechanism must address

The PP mechanism not only intends to deal with users who will intentionally lie, it must also deal with users who are uninterested in participating, users who behave randomly, and users who may collude to get a payout either inside or outside of the mechanism.

What kind of contexts affect the PP mechanism?

The PP mechanism needs to be designed to fit the context. Thus, there are many different PP mechanisms depending on the type of context such as the number of tasks and participants, the heterogeneity of tasks and participants, the number and types of signals (e.g., yes/no, 1–5 scale).

What are the applications of the PP mechanism?

Areas where PP mechanism is applicable include without limitation:

• surveys (i.e., crowdsourcing),
• peer assessment,
• product reviews, and
• determining the offensiveness of an article.

Specific areas we are interested in include blockchain, education, citizen science, and academia.

Blockchain

The PP mechanism can be used to help determine consensus on information fed to data oracles and to shore up token curated registries. Thus, making it easier for DeFi protocols to operate and reduce instances of malfeasance associated with data oracle manipulation.

Education

The PP mechanism can be used for peer assessment in a Massively Open Online Course (MOOC) where students in a class grade each other’s assignments.

Citizen Science

The PP mechanism can be used as an incentivization mechanism in citizen science projects so that citizens can assess and validate each others work (e.g., citizens labeling images and receiving a payout if their labeling has a high correlation with another citizen’s image label).

Academia

The PP mechanism can be used for peer review of academic articles.

Peer Production Communities

The PP mechanism can also be very helpful for peer production communities because peer production communities emphasize peer processes in the production of their outputs.

Help us investigate the PP mechanism

If you enjoyed this article or podcast on the PP mechanism and you want to know more, please see the Supplementary Material section.

Additionally, please consider joining us (as a member or contributor) and investigating the possible applications of the PP mechanism.

Supplementary Material

Readings

1. Peer Prediction with Heterogeneous Users
2. Informed Truthfulness in Multi-Task Peer Prediction
3. Paying for the Truth: The Efficacy of a Peer Prediction Mechanism in the Field
4. An Information Theoretic Framework For Designing Information Elicitation Mechanisms That Reward Truth-telling
5. Token-Curated Registry with Citation Graph
6. Information Diffusion Enhanced by Multi-Task Peer Prediction
7. Token-Weighted Crowdsourcing
8. Incentivizing evaluation with peer prediction and limited access to ground truth
9. A Robust Bayesian Truth Serum for Non-Binary Signals

About LedgerbackØDCRC

Established in 2018, the Ledgerback Digital Commons Research Cooperative (LedgerbackØDCRC) is a nonprofit cooperative association and distributed p2p network for unifying the study of the internet and society and fostering collaboration between stakeholders to advance towards a global technological commonwealth.

Our research approach is an inter/cross-disciplinary approach, with the goal to eventually employ an anti/antedisciplinary approach as we continue to grow.

Global Technological Commonwealth

A global technological commonwealth (as summarized here) is a sociotechnical imaginary (i.e., a vision of the future) that “consists of post-capitalist society where communities of mutual interest cooperate in the construction of institutions of regenerative economic relations” [1]. The technological design principles include:

• “incorporating planetary boundaries,
• modelling on natural biological ecosystems,
• enabling the redefinition of value,
• enabling radically democratic coordination and governance, and
• allowing for the growth of a cooperative commons as the desirable future” [1].

For more information on the global technological commonwealth (and to get some background), we recommend reading Dr. Sarah Manski’s article, Distributed Ledger Technologies, Value Accounting, and the Self Sovereign Identity.

Areas of Interest

Our areas of interest include, without limitation:

1. Web 3 technologies (blockchain, pubs, secure scuttlebutt, fediverse, smart contracts, etc.)
2. Collaborative economy (platform ecosystems, business models, platform capitalism, platform cooperativism, ownership economy, p2p/commons, digital labor, social contracts, etc.)
3. Future of work (open value accounting, peer production, self-management practices, digital organizations, etc.)
4. Digital Infrastructure (internet service providers, hardware, mesh networks, machine-to-machine economy, Internet-of-Things, etc.)
5. Data science and ethical AI (AI/ML, human-in-the-loop AI, data analytics, algorithmic policy, algorithmic governance, etc.)
6. Information privacy and security (data stewardship, cybersecurity, privacy-by-design, zero knowledge proof, cryptography, etc.)
7. Knowledge Commons (notetaking tools, knowledge repositories, decision-making models, decision analysis, collective intelligence, swarm intelligence, EdTech, etc.)
8. Metascience (open science, citizen science, science funding, bibliometrics, publishing, etc.)
9. Personal Data or Digital Identity economy (data stewardship, data monetization, self-sovereign digital identity, decentralized identifiers, digital identity, decentralized identity, data privacy, data cooperatives, data trusts, etc.)
10. Open Finance (e.g., alternative currencies, timebanking, community currencies, decentralized finance, prize-linked savings accounts)
11. Complex systems (game theory, mechanism design, dynamic systems, simulation, etc.)
12. Cryptoeconomics (bonding curves, cryptoprimitives, complex systems, peer prediction, schelling points, tokenization, etc.)
13. Sustainability (circular economy, renewable energy, community-owned utilities, etc.)
14. Science and Technology (how science and technology interact with society positively and negatively, and how the relationship between them can be changed for the social good)

Problem Statements

Some example problem statements we are investigating are described in the following articles:

• Creating an alternative ed-tech by Fathom
• Local-first software: You own your data, in spite of the cloud by Inkandswitch
• A Preliminary Review of Blockchain in the Music Industry by LedgerbackØDCRC
• OpenStreetMap is Having a Moment by Joe Morrison
• AI needs to face up to its invisible-worker problem by Will Douglas Heaven

Currently, LedgerbackØDCRC is run by volunteers (and we thank them for all their effort!).

Membership Benefits

Our primary stakeholders or intended beneficiares of our membership are investigators (scholars, researchers, academics, activists, makers, technologists, etc.), practitioners, citizens, and our staff (the people who make LedgerbackØDCRC run!) .

The benefits we provide or plan to provide to our members includes:

1. online portal (email included)
2. cloud infrastructure and interactive computing infrastructure
3. combining resources
4. mapping, data analytics, knowledge tools
5. grantwriting support
6. fundraising support
7. sharing experiences
8. publishing support
9. research assistance
10. networking offline and online
11. informing members of opportunities
12. providing resources

We do not have a membership fee (no need to pay $2,500.00 to join our community) but we do have annual fees ($50.00/year or provide 40 hours of time to cooperative-directed activities) to keep the cooperative operational.

Join us via the form below or send an email to ledgerback@gmail.com.

Describing LedgerbackØDCRC

The LedgerbackØDCRC is best understood as multi-purpose cooperative (we don’t fall neatly into a category 😖) that can better be described by its functions (or really a mix of a foundation, ecosystem and a research institute):

1. Research Institute: We produce original research (basic, applied, empirical) and analyses on the internet and society, formulate models, tools, and designs and practices, grow a body of knowledge on the internet and society with an emphasis on how to transition towards a global technological commonwealth, develop prototypes, open source software and proof-of-concepts, and run citizen science projects.
2. Data Cooperative: We produce and analyze datasets, trends, and other areas of interest by collecting publicly available data or curating data from our members or participants in our projects, and offer our analyses and datasets to the general public and interested parties.
3. Foundation: we support efforts to advance towards a global technological commonwealth, hosting events and workshops, hosting distributed communities, and acting as a host for the greater Ledgerback ecosystem.
4. Observatory: We monitor progress among the many sociotechnical ecosystems
5. Academy: We produce open source educational materials and help others find and take courses on the internet and society, and develop the skills needed to cause transformational change towards a global technological commonwealth.
6. Distributed community: We work together with people all across the world online to build a knowledge commons and provide resources to those who need them.

Supporting the LedgerbackØDCRC

You can support us in many different ways including:

1. Becoming a member (best way to do it)
2. Making a direct donation (donations are not tax deductible for now but we are working on it!)
3. Getting involved with one of our cooperative research projects as an investigator or citizen
4. Volunteering to help take on our core or administrative activities
5. Connecting the LedgerbackØDCRC with other individuals and organizations working in similar areas
6. Sending us your feedback on our articles, podcasts, and other media
7. Mentioning to others who we are and what we are doing
8. Taking our ideas and models and putting them into practice (with attribution 😆)
9. Send us a message asking for problems to solve.