A deepdive into FIL-RetroPGF-1 results
- Infrastructure & Dependencies
- Tooling & Utilities
- Education & Outreach
- Protocol Research & Development
- Collective Governance
- Products & End User UX
- Other
These categories encompass the range of Public Goods needed to support a thriving and growing Filecoin network. By distributing 200k FIL to projects in these categories in the first round, FIL-RetroPGF funds established projects and seeds the formation of new projects that support the Filecoin mission to create a decentralized, efficient, and robust foundation for humanity’s information.
After badgeholders voted, the round organizers anonymized, collated, and analyzed the results. Let’s dive into these results now.
Project Funding Breakdown
Fig. 1 shows the percentage of the total funding allocated to each project. Projects which did not receive funding are not shown.
The top 5 projects to receive funding were:
- GLIF Nodes & RPC API — 4365 FIL
- Lotus Dependencies — 4315 FIL
- DRAND — 4186 FIL
- Lighthouse.Storage — 4018 FIL
- Filecoin Orbit Ambassadors —3795 FIL
These projects captured ~10.5% of the total funding available in Round 1!
The mean and median projects were allocated 1860 FIL and 1883 FIL, and the top scoring project was allocated 4635 FIL!
Se here for a full results breakdown.
Recall that projects were nominated by category. Fig 2 shows how votes and funding were allocated across the different categories. In aggregate, most votes and funding went towards projects in the Infrastructure and Dependencies category. The average funding an Infrastructure and Dependencies project received was 40.73% greater than the average funding received per project across all categories! This suggests that the badgeholders felt that the Infrastructure and Dependencies projects had the most impact on the ecosystem. Protocol R&D and Governance are other categories in which the average project in that category received more funding than the average (6.1% and 9.6%, respectively), whereas the other categories received less funding than the average project.
Ballots Cast
The most common number of ballots for a project to receive was 8, and the minimum was 3, suggesting generally good coverage across the projects from the Badgeholders in terms of expertise to express an opinion and vote. 94% of the projects that submitted applications were eligible for funding because they met the quorum requirement (at least 5 ballots cast). See Fig 3.
Fig 4 shows the ballots cast by the badgeholders for each project. It is sorted by the number of ballots each project received, which does not necessarily correspond to the funding received by each project shown in Fig 2. The top 5 projects which received the most number of ballots are:
- DRAND — 19
- Asia SPWG — 16
- Filecoin Proofs — 16
- Beryx — 16
- Banyan — 16
Only one of the top five projects that received the most ballots (DRAND) was also in the top five list of projects that received funding.
Fig 5a shows how the funding was distributed across the different projects. Fig 5b shows a rank ordering of projects and the total percentage of funding that they received. The top 4.72% projects received 10% of the total allocation, the top 12.26% projects received 25% of the total allocation, the top 29.25% projects received 50% of the total funding, the top 50.94% of projects received 75% of the total funding, and the top 93.4% of projects received 100% of the funding.
Badgeholder Voting Patterns
Recall that each badgeholder had 100 votes that they could allocate across all the projects, with a maximum being 16. Fig 6 plots how many votes badgeholders allocated to a given project. We observe that the most common vote is 1, with the second most common being 0. Higher votes, such as 16 / project, were given out more sparingly.
For discussion purposes, let us define the “temperature” of a badgeholder to be related to the number of ballots that the badgeholder cast. A cold badgeholder is one that distributes their votes across many ballots. Conversely, a hot badgeholder is one that concentrates their votes on a small number of projects. Using this framework, we can rank badgeholders by their “temperature” and observe how they voted across the various projects (Fig 7). Using a ranking of badgeholders in the FIL-RetroPGF-1 cohort, the top 10% of cold badgeholders are Badgeholders 0–2 and the top 10% of hot badgeholders are Badgeholders 22–24.
How reliable is the size of the badgeholder set in a statistical sense?
How confident are we in the distribution of funds indicated in Fig 5? We performed a bootstrap analysis by selecting all possible subsets of ballots cast by badgeholders, with a minimum number of 20 badgeholders and a maximum of 25. This creates 68000 samples of badgeholder votes. We then compute the confidence intervals of these distributions and overlay them with the distribution that will be used for funds distribution. This is shown below in Fig 8. This illustrates two points: i) IQR indicates that having 25 badgeholders is reasonable at estimating the true signal since the confidence intervals shown from different partitions of badgeholders are not very dispersed, ii) As the number of badgeholders decreases in the bootstrapped sample, the number of projects which receive funding is reduced, because it becomes harder to meet quorum with fewer badgeholders.
Counterfactuals
Considering the results above, in this section, we present how the allocation distribution could have been different, had different scoring rules been applied.
C1 — What if quorum meant strictly positive votes?
A project was eligible to receive funds if it received at least 5 ballots. In the official scoring methodology, 0 votes were counted in the quorum. What would have happened if zero votes were not counted — that is, a project needed to receive 5 votes > 0 rather than 5 votes >= 0? Fig 9 shows how the funds would have been distributed for these rules. If the quorum needed to be strictly positive, an additional 24 projects would not have been eligible to receive funding. Had this rule been used, the remaining projects that were eligible for funding would have received more since all funding is redistributed to a smaller number of projects.
C2 — What would the distribution look like if quorum was not 5?
Fig 10 shows how the funds would have been distributed for different quorum requirements. In general, as the quorum requirement increases, the number of projects eligible for funding decreases, and the remainder of eligible projects receive more. From this, we see that the choice of quorum is a tradeoff between rewarding highly impactful projects more generously and distributing them across a larger set of projects that had a less subjective impact (as assessed by the badgeholders).
C3 — What if we used median or quadratic scoring rather than mean?
In the Voting process section, we described that votes from different badgeholders are aggregated using the mean function. What if we had used the median or a quadratic function? What would the distribution have looked like? Fig 11 compares the hypothetical allocations when using the alternative score aggregation functions. Mean and Quadratic aggregation functions result in smooth distributions that fund more projects, while a median aggregation function results in a piecewise distribution that results in less funded projects. The behavior of the median aggregation function is expected because of the method by which voting was set up. Because badgeholders had 100 votes to distribute in integer quantities, this results in the median snapping to discrete intervals, while mean and quadratic do not. Additionally, Figure 4 shows that several projects received a majority of zero votes, so the median distribution profile cuts off more projects from receiving funding.
C4 — What if we enforced a minimum amount of distribution?
In FIL-RetroPGF-1, the round decided not to implement a minimum amount of funds to be distributed, to see a long tail and maximise outreach to many projects.
In OP however, projects not only needed to meet quorum, but also needed to meet a minimum distribution of funds in order to be eligible for funding. Fig 12 shows hypothetical distributions of funds to projects, had a minimum distribution of 500 FIL or 1000 FIL been implemented. Increasing the minimum distribution has the same general effect pattern as increasing the quorum requirement, as seen in Fig 10.
C5 — What if we removed the top 10% of hot and cold badgeholders?
Recall that we previously introduced the notion of the temperature of a badgeholder. What if we had removed the top 10% of cold badgeholders (who voted across many projects), and the top 10% of hot badgeholders (who voted across the fewest projects)? How would the distribution of funds change?
Fig 13 indicates that removing the top 10% of hot badgeholders results in a very similar allocation distribution to the actual distribution. Conversely, removing the top 10% of cold badgeholders results in an outsized reduction in the number of projects funded. The reasons are:
- Because the cold badgeholders helped more projects meet the quorum requirement, so funds became more spread out.
- The coldest badgeholders adding 0, 1 or 2, typically lowered the mean score of the project, making the distribution of funds more spread out again.
- Adding votes to low scoring projects takes some funds away from higher scoring ones.
Effect of UX on Votes and Distribution
Badgeholders used the easy-retropgf software to cast their ballots. This interface presented projects in alphabetical order to badgeholders. Did this ordering have a statistically significant effect on the amount of funding a project received?
To understand this, we indexed projects by the starting letter and computed a regression between the project index and a) the total number of ballots cast for the project and b) the mean score per project. Fig 14 a and b indicate that there is a small relationship. A correlation of -0.27 is observed between the project’s starting letter and the number of ballots cast for the project, meaning that projects that started with letters occurring later in the alphabet had fewer votes cast for them. Similarly, a correlation of -0.20 is observed between the project’s starting letter and the mean score of the project.
MCMC techniques revealed that this effect was statistically significant. More precisely, we found that the true correlation between a project’s starting letter and its final score was between -0.34 and -0.03, with a 95% confidence interval. This translates to an average funding decrease of approximately 40 FIL for every letter in the alphabet, shown below in Fig 15.
Conclusion
FIL-RetroPGF-1 was the first iteration of a new mechanism to fund public goods and encourage new innovations within the Filecoin ecosystem. In the first round, 200k FIL was distributed to 99 projects in seven categories. The category receiving the most funding was Infrastructure & Dependencies, which also received the most amount of funding per project on average! All other categories received funding within 10% of the overall average. The top 5 projects captured ~10.5% of the total funding available!
Counterfactuals indicate that the mechanism chosen to convert votes to allocation resulted in the kind of fund distribution that many of those involved in the first iteration of the round wanted to see. Specifically, the quorum threshold of 5 and no minimum score requirement resulted in 94% of projects receiving funding. The chosen score function (mean) also resulted in a smooth distribution of funds to projects compared to the median (Fig 11).
On an operational note, the organizers regret the miscommunication with badgeholders regarding the quorum. The question was whether a zero vote should be included in the quorum. The example in the badgeholder manual indicates that a zero vote should not count towards a quorum — this was an error on the part of the organizers. The philosophical reason behind counting a zero vote in a quorum is that the purpose of a quorum is to receive enough votes to have enough confidence in the scoring function (mean) and not a judgment on whether the project should receive a distribution. The counterfactual in Fig 9 indicates that had only strictly positive votes counted towards quorum, an additional 24 projects would not be eligible for funding.
Secondly, Fig 14 and 15 and the relevant analysis indicate a statistically significant correlation between the letter that the project started with and the final distribution the project received. While small, we will explore UI upgrades to reduce these effects in future rounds.
We thank the Filecoin ecosystem, nominators, applicants, and badgeholders for the critical role they played in making FIL-RetroPGF-1 successful. Be on the look out for FIL-RetroPGF-2!!
