Since the launch of our first pilot program (Block I) in Sept 2021, 39 fellows have graduated, 9 resident auditors have been onboarded, and 20 codebases have been audited.
We now have the bandwidth to be auditing codebases both during and outside of our fellowship programs.
This is a retrospective into our 2nd and 3rd fellowships that took place in Q1/Q2 2022. For our first pilot fellowship retrospective, please see pilot retrospective.
Overview (the tl;dr)
Block II ran from Feb 2, 2022 to Mar 11, 2022.
Block III from June 6, 2022 to July 14, 2022.
Security Review Reports
The security reviews produced by the Block II & III fellowship:
Flash Feedback
Evolution of the Block
The core structure of a Block was largely unchanged from the Pilot and the intro blog post.
However, we continued iterating on a few aspects with the aim of improving fellow commitment, engagement, and experience.
Both Block II and III had the typical structure of:
- A kickoff meeting and warmup week (week 0)
- Four weeks of reviewing smart contracts (weeks 1–4)
- A closing ceremony
- Guest Speakers on most Fridays
However, Block II and III differed drastically in aspects such as:
- Pre-block interview process
- Structure of fellow + resident teams
- Coordination and communication
To further outline the evolution of the block, the following sections will detail a short summary of Block II, and a longer section of Block III taking a deeper dive into the aspects described above.
Block II
Pre-block
We entered the block having learned the following lessons from the Pilot (Block I):
- Fellows work better in larger groups than individually
- Adding a competitive nature to the reviews engaged the fellows more
- Peer-to-peer sharing of learnings led to a better experience for more of the fellows
This led to us focusing less on fixed approaches and experimenting with a more data-driven, feedback-orientated approach. We explored variables such as group size, types of collaboration, presentation styles, group voting vs leadership, etc. This effectively relied on the participants steering the blocks into a better direction, and having them reinforce learning regarding what was going well, and not so well.
Post-block
After our Block II post-mortem, the primary concern was a high variance of fellow engagement. 1 out of 3 teams had consistent meetings throughout the weeks to discuss findings. Other teams started strong but engagement dwindled down as weeks went by.
Here were the main takeaways from Block II:
- Teams may have been too small
- yAcademy structure & communication was too hands-off
- Pacing was too fast for review work
- Interview process criteria weighed too highly on technical ability; needed balance on commitment, teamwork, and availability
Block III
The yAcademy core team had grown from Block II. We had a few more contributors and residents — this gave rise to the ability for more evenly-spread work. We also began stressing more effective internal communication and coordination.
Interview Process
For comparison, here were the interview process pipelines for Blocks II & III:
- Block II — application -> fit & tech interview -> offer
- Block III — application -> remote technical assignment or quiz -> fit & tech interview -> offer
Block III introduced the intermediate step of remote technical assignment or quiz. Because yAcademy is not a beginner-friendly program, combined with the higher volume of applicants, and limited availability of the yAcademy’s core team, we needed a method to determine a candidate’s seriousness and baseline knowledge level before proceeding with the fit & tech interview.
Block Structure
For contrast, here was the structure of the previous block (Block II):
- 4 different contracts; 1 contract/week
- Teams of 3–4 fellows, separately reviewing contracts and forming their own team report.
Block III changed the format drastically as we had more experience, and helping hands (Residents and Operations):
- 4 different contracts; Fellows exposed to 3 contracts
- In the first half, Fellows were split into two large groups, under the guidance of a Resident. They were exposed to 1 contract for 2 weeks.
- In the second half, Fellows were one big group under the guidance of a Resident. They were exposed to 2 contracts in 2 weeks (1 per week).
The first half was more well-received than the second half with the main criticism being that one week was too short for a proper review.
Communication and Coordination
We are a technical group of doers, with our hearts revolving around the engineering and security part of the process. We do our best to remain nimble, minimize supervision, and over-coordinating. However, we realized that this was not an excuse for a lack of effective communication.
Given this, the yAcademy core team stressed better internal and external communication to minimize confusion during the block. Though we seem to be veering towards a positive trend, this remains one of our primary pain points (further discussion under the Improvement section).
Feedback
10 of the 14 graduate fellows provided feedback for Block III. In general, the block was very well-received:
Web3 knowledge increase
There was general improvement across the board, with only 20% of participants reporting no improvement. The highest rate of improvement was seen with users who rated their original understanding at or below average.
Of note, fellows that reported no improvement already started out at an above average web3 knowledge.
Web3 security knowledge increase
Markedly strong improvement across the board. Again, 20% reported no improvement, but on average, the reported increase is higher than general web3 knowledge and seems to follow the same trend — the less the fellow rated themselves pre-block, the higher they rated themselves post-block.
Of note, the average understanding of web3 security pre-block was much below that of general web3 knowledge.
First vs Second Half
As described in Block Structure, the first half of the block was generally more well-received than the second half. Reasons included:
- One week being too short for a thorough security review
- Fatigue from first half; no breaks
- Difficult contract in 2nd half
- Schedule conflicts
Trending well
Based on feedback, growth metrics, and observations, we see yAcademy scaling, and trending in the positive direction. We continue to encourage the learning, teaching, sharing of web3 security knowledge, and collaboration across open-source. We are also proud to have met, and brought together so many enthusiastic and intellectually-curious minds across the globe who remain kind, open, and respectful. At the end of the day, we are just engineers & scientists who wish to grow a safer ecosystem.
Improvement
As mentioned in Communication and Coordination, our team has a heavy technical bias. Though we continue learning and iterating, the non-technical aspects required for yAcademy to succeed are still under heavy duress. Operations, communication, coordination, will continue to be a focal point as we begin traveling the road of self-sustainability.
We will also continue experimenting with parameters such as interview process, block format, and onboarding resources, to enhance fellowship experience.
What’s next
We are heartened by the ever increasing number of applicants to our fellowship programs, and will begin preparation for the next one in the coming weeks. We ask those who applied to be patient as we ramp up our internal ops/comms/process to effectively deal with the higher load of applicants, onboarded residents, and the audit requests we receive.
Meanwhile, our core and resident auditors continue to receive and take on audit requests from various projects, many of which are returning clients. That work is focused and is not affected by the admin work to organize and run the fellowships. Stay tuned for updates!
