Cent’s New Sorting System

Introducing the Iterative Comparison Algorithm (ICA)

Right now on Cent, users ask questions with bounties in ETH that incentivize responses from the network. Users vote on the best responses (earning ETH for doing so) on a six-point scale from -3 to +3. We went with this curation approach at the time because it seemed like the most accurate way to capture the sentiment of the network on the quality of varying answers.

As the network has grown, however, it’s become increasingly cumbersome for users to deal with the volume of answers the network produces. Bounties regularly garner 100+ responses, and asking users to individually give each and every one an objective grade seemed like an increasingly difficult ask.

So, we went back to the drawing board. What really matters when determining the quality of answers? What does the word quality even mean? Is voting with a rating of -3 to +3 really the most accurate way of capturing how valuable something is? Is voting even necessary?

We realized that quality derives its meaning based on the comparison between things. It’s difficult to assign objective levels of value, yet easy to assign relative levels of value.

How much quality does a Rolls-Royce have? I don’t know what that means. But which is higher in quality — a Rolls-Royce or a Kia? That I know.

Via comparison, we are able to make judgments much more confidently, quickly, and accurately. So rather than have a voting scale that requires users to grade on a global scale, we created the Iterative Comparison Algorithm — or ICA. The ICA pairs answers with each other, simply asking the user a single question each time: which is better?

After coming up with this solution, we realized that this type of sorting has been discovered before, and is sometimes known as “pairwise comparison”. As an academic paper from Johns Hopkins states, there are many reasons pairwise comparisons are more powerful than other methods:

Pairwise comparison can be greedy because the instrument can easily present as many (or as few) prompts as each respondent is willing to answer.

New items contributed by respondents can easily be integrated into the choice sets of future respondents, enabling the instrument to be collaborative.

Finally, pairwise comparison can be adaptive because the pairs to be presented can be selected to maximize learning given previous responses.

These properties exist because pairwise comparisons are both granular and modular; that is, the unit of contribution is small and can be readily aggregated.

Additionally, the paper outlines some practical benefits:

Pairwise comparison makes manipulation, or “gaming,” of results difficult because respondents cannot choose which pairs they will see; instead, this choice is made by the instrument. Thus, when there is a large number of possible items, a respondent would have to respond many times in order to be presented with the item that she wishes to “vote up” (or “vote down”).

Second, pairwise comparison requires respondents to prioritize items — that is, because the respondent must select one of two discrete answer choices from each pair, she is prevented from simply saying that she likes (or dislikes) every option equally strongly.

Finally, responding to a series of pairwise comparisons is reasonably enjoyable, a common characteristic of many successful web-based social research projects.

Our ICA leverages these benefits by progressively judging the best answers against the best answers (identifying the great) and the worst answers against the worst (identifying the terrible). This shift evolves curation on Cent bounties from a voting mindset, to a sorting mindset. This affects the user experience in a way that we feel is a lot more fun, quick, and easy. And as always, each pair you sort earns you a bit of ETH.

Every time a user returns to an open bounty, they’re presented with new pairings of answers that are novel, rather than having to go through all the answers they’ve already voted on to dash off some votes on whatever has come in most recently. We’ve also removed the @names and comments from answers until after you’ve sorted them, to keep the sorting as fair as possible.

The ICA and new sorting UI is live now — head on over to Cent and check it out.