Reflections on Quadratic Voting as a Hackathon Judging Mechanism
Last week we concluded the second edition of Codeless Conduct, which took place from February 23rd to March 11th. We are extremely humbled by the turnout and engagement in this edition. The hackathon ended with over 1200 participants and 630 bounty submissions 🎉
In this latest round, we aimed to not only make the experience better by increasing the amount of bounties and time available, but also tried out a new judging mechanism for our Open Track bounties. Specifically, we experimented with quadratic voting.
This article is a retrospective on the quadratic voting process as a hackathon judging mechanism. We dive into:
- Challenges with judging open-ended bounties
- How quadratic voting helps us address those challenges
- How we implemented QV at Codeless Conduct
- General Takeaways & What we could try for next time
Codeless Conduct Overview
Codeless Conduct is a no-code hackathon series aimed to onboard non-technical contributors into web3. Our mission is to be top-of-funnel for fresh talent to discover opportunities and make their first contributions to existing projects in the industry.
The event offers workshops by industry mentors in the fields of design, community building, marketing, data/analytics, tokenomics, and governance. These were then paired with related bounties for participants to tackle.
There are 2 types of bounties:
- Sponsor Bounties are designed and paid for by our sponsors. Submissions are judged internally by the respective projects. Depending on the sponsor and bounty, it could be winner-take-all or graduated payouts made to 1st-2nd-3rd place. AKA the “classic” way hackathon and bounties winners are typically selected.
- Open Track Bounties are designed to give participants an opportunity to create or propose something to projects of their choice. The tracks available were design, community building, marketing, data analytics, tokenomics, governance, and general.
Within each track, we limited the variability of submissions by providing options of tasks participants can work on.
Open Track Judging Challenges
Since Open Track bounties are more open ended in terms of scope, and required comparing apples to oranges, we looked for a way to ensure that judging was somewhat decentralized, representing more voices than just the Codeless Conduct team.
In the first edition of Codeless Conduct, we allowed everyone in the community to vote using Discord reacts. With this mechanism, each participant was allowed to vote on any number of projects once (including their own). The prize pool was split evenly among 5 submissions which received the highest number of “votes”.
This method, though community-driven, had several shortcomings:
- Though participants could vote on as many submissions as they’d like, there was no way to express degree of preference i.e. how much they liked one submission relative to the others.
- Voting was open to everyone in our Discord, so there was no way to gauge whether voters were actually engaged in the event and align these them on the judging criteria properly.
- The “equal split for top 5 winners” was arbitrarily decided. There could have been a submission amongst the top 5 that was significantly higher quality than the rest, or the 6th place could have also been a great submission that missed the cutoff by one vote and got nothing.
So we looked for ways to improve the judging process for open track bounties…
Enter quadratic voting
Quadratic Voting is a collective decision-making methodology that allows participants to not only cast votes for/against an issue, but express the intensity of their preferences.
Each participant is given a number of “voice credits” that can be used to vote for a submission. The first vote costs 1 credit, but each additional vote incurs a quadratically increasing cost. For example, the second vote costs 4 credits, the third vote 9 credits, and so on following this formula:
Cost to the vote = (number of votes)²
Quadratic Voting has been used to decide on proposed spending bills by the Colorado State House of Representatives, as well as to determine winners for Taiwan’s Presidential Hackathon of 2019. The hackathon was global initiative to bring innovations for social needs and public contracting for the country. Different stakeholders could submit their proposals and projects, which were then judged by millions of citizens participating in quadratic voting.
QV addressed the above issues for Open Track judging by:
- Allowing voters to express degree of preference. While voters are increasing the chances of submissions winning with each additional vote, the quadratic nature ensures that only those who really care about certain submissions will cast additional votes for them — because they are doing so at the expense of their ability to vote on other submissions.
- Providing a justifiable way to split the prize pool. Using the vote counts and calculating the % of votes each submission received, we could split the prize pool pro-rata, without having to do winner-take-all or arbitrary splits.
How we implemented quadratic voting
1. Select judging panel
For each Open Track (7 in total), we had a judging panel consisting of 3 members of the Codeless Conduct team and 2 community members — making up 35 judges overall.
Community judges were volunteers, with the only criteria being that they must not have had a submission for the track they were judging for.
2. Create ballots and generate personalised voting links
We imported bounty submissions into Radical Exchange’s open source quadratic voting tool, creating a separate “event” for each track.
Each judge was allocated 99 vote credits that they can assign to submissions they deemed to be best per the judging guidelines.
3. Export results
After all judges have submitted their votes, the QV tool produced a results breakdown ordered by number of votes received.
4. Calculate payouts
We imported results into a spreadsheet and performed the following calculation:
- Tally all the votes to get the total vote
- Calculate % of votes received by each submission
- Exclude those with <5% of the votes and recalculate
- Multiply new % of votes with the prize pool
The prize pool was then split amongst winners pro-rata to the amount of votes they received.
Takeaways
- The pro-rata payout splits was an effective way to determine how to allocate the prize pool such that more participants are rewarded for their work.
- The 5% cutoff was important, as otherwise we would end up paying out unfeasibly low amounts to “long-tail” projects, taking away from top submissions. 5% was chosen as it is the threshold for most democratic multi-party governmental parliaments, however we could iterate on this depending on the number of submissions that came in for each track. We could increase the threshold to ensure top submissions had a higher payout and not be too “diluted” if there are many submissions for that track, for example.
- We should add more stakeholders to the judging panel for each track. For example in addition to Codeless Conduct team and community judges, invite industry professionals in the relevant tracks. QV has been effective for governmental decisions because it protects minority interests by allowing them to vote on less popular topics that they care strongly about. By including more diverse voices in the judging process, the outcome will be more representative of community preferences as a whole.
- There’s great opportunity for tooling and educational resources to be improved to encourage more experiments with QV. We’d love to see an integration with hackathon platforms like Gitcoin or Devpost. But in the meantime, these can be good bounties for participants to tackle in future versions of Codeless Conduct!
While QV is still a relatively niche mechanism, it offers a way in which community members can express how they feel about an issue in addition to the strength of their preferences. The concept takes a bit of time to grasp, but once understood, can encourage community members to be more engaged in the decision making process as they can express their preferences in a more meaningful way.
What’s next for Codeless Conduct
Thank you again to all those who participated this round of Codeless Conduct. We certainly learned a lot from the organising side and can’t wait to deliver a better experience this time. If you’re interested in helping put on the next edition of Codeless Conduct, please reach out!
- 💸 Donate to our Grant on Gitcoin
- 📰 Follow us on Twitter to get the latest updates
- 👾 Join our Discord to meet the community
If you would like to sponsor a future edition of CC, please reach out to us at sponsors@codelessconduct.org.
This event would not have been possible without our amazing sponsors who have provided over 40 bounties and offered over $30k in prizes. A huge thank you to Gnosis Safe, Connext, Fluence, DXdao, Tally, Covalent, Nexus Mutual, B Protocol, and Forefront for your support. Also thanks to the team at Gitcoin for hosting the event and operational help. Special shoutout to Leon Erichsen for his encouragement for us to explore quadratic voting and for reviewing drafts of this post.
