Blocks II & III Retrospective

thraull
yAcademy
Published in
6 min readAug 23, 2022

Since the launch of our first pilot program (Block I) in Sept 2021, 39 fellows have graduated, 9 resident auditors have been onboarded, and 20 codebases have been audited.

We now have the bandwidth to be auditing codebases both during and outside of our fellowship programs.

This is a retrospective into our 2nd and 3rd fellowships that took place in Q1/Q2 2022. For our first pilot fellowship retrospective, please see pilot retrospective.

Overview (the tl;dr)

Block II ran from Feb 2, 2022 to Mar 11, 2022.

Block III from June 6, 2022 to July 14, 2022.

Quick statistical comparison across blocks.

Security Review Reports

The security reviews produced by the Block II & III fellowship:

Block II

Block III

Flash Feedback

Fellows comfort levels performing web3 security reviews pre- and post-block (more detail in Feedback section)

Evolution of the Block

The core structure of a Block was largely unchanged from the Pilot and the intro blog post.

However, we continued iterating on a few aspects with the aim of improving fellow commitment, engagement, and experience.

Both Block II and III had the typical structure of:

  1. A kickoff meeting and warmup week (week 0)
  2. Four weeks of reviewing smart contracts (weeks 1–4)
  3. A closing ceremony
  4. Guest Speakers on most Fridays
Refresher: The typical block structure as depicted in Introducing yAcademy

However, Block II and III differed drastically in aspects such as:

  • Pre-block interview process
  • Structure of fellow + resident teams
  • Coordination and communication

To further outline the evolution of the block, the following sections will detail a short summary of Block II, and a longer section of Block III taking a deeper dive into the aspects described above.

Block II

Pre-block

We entered the block having learned the following lessons from the Pilot (Block I):

  • Fellows work better in larger groups than individually
  • Adding a competitive nature to the reviews engaged the fellows more
  • Peer-to-peer sharing of learnings led to a better experience for more of the fellows

This led to us focusing less on fixed approaches and experimenting with a more data-driven, feedback-orientated approach. We explored variables such as group size, types of collaboration, presentation styles, group voting vs leadership, etc. This effectively relied on the participants steering the blocks into a better direction, and having them reinforce learning regarding what was going well, and not so well.

Post-block

After our Block II post-mortem, the primary concern was a high variance of fellow engagement. 1 out of 3 teams had consistent meetings throughout the weeks to discuss findings. Other teams started strong but engagement dwindled down as weeks went by.

Here were the main takeaways from Block II:

  • Teams may have been too small
  • yAcademy structure & communication was too hands-off
  • Pacing was too fast for review work
  • Interview process criteria weighed too highly on technical ability; needed balance on commitment, teamwork, and availability

Block III

The yAcademy core team had grown from Block II. We had a few more contributors and residents — this gave rise to the ability for more evenly-spread work. We also began stressing more effective internal communication and coordination.

Interview Process

For comparison, here were the interview process pipelines for Blocks II & III:

  • Block II — application -> fit & tech interview -> offer
  • Block III — application -> remote technical assignment or quiz -> fit & tech interview -> offer

Block III introduced the intermediate step of remote technical assignment or quiz. Because yAcademy is not a beginner-friendly program, combined with the higher volume of applicants, and limited availability of the yAcademy’s core team, we needed a method to determine a candidate’s seriousness and baseline knowledge level before proceeding with the fit & tech interview.

Block Structure

For contrast, here was the structure of the previous block (Block II):

  • 4 different contracts; 1 contract/week
  • Teams of 3–4 fellows, separately reviewing contracts and forming their own team report.

Block III changed the format drastically as we had more experience, and helping hands (Residents and Operations):

  • 4 different contracts; Fellows exposed to 3 contracts
  • In the first half, Fellows were split into two large groups, under the guidance of a Resident. They were exposed to 1 contract for 2 weeks.
  • In the second half, Fellows were one big group under the guidance of a Resident. They were exposed to 2 contracts in 2 weeks (1 per week).
Block II vs Block III structures. Fx refers to Fellow groups while Rx refers to Resident groups.

The first half was more well-received than the second half with the main criticism being that one week was too short for a proper review.

Communication and Coordination

We are a technical group of doers, with our hearts revolving around the engineering and security part of the process. We do our best to remain nimble, minimize supervision, and over-coordinating. However, we realized that this was not an excuse for a lack of effective communication.

Given this, the yAcademy core team stressed better internal and external communication to minimize confusion during the block. Though we seem to be veering towards a positive trend, this remains one of our primary pain points (further discussion under the Improvement section).

Feedback

10 of the 14 graduate fellows provided feedback for Block III. In general, the block was very well-received:

Web3 knowledge increase

There was general improvement across the board, with only 20% of participants reporting no improvement. The highest rate of improvement was seen with users who rated their original understanding at or below average.

Of note, fellows that reported no improvement already started out at an above average web3 knowledge.

Web3 security knowledge increase

Markedly strong improvement across the board. Again, 20% reported no improvement, but on average, the reported increase is higher than general web3 knowledge and seems to follow the same trend — the less the fellow rated themselves pre-block, the higher they rated themselves post-block.

Of note, the average understanding of web3 security pre-block was much below that of general web3 knowledge.

First vs Second Half

As described in Block Structure, the first half of the block was generally more well-received than the second half. Reasons included:

  • One week being too short for a thorough security review
  • Fatigue from first half; no breaks
  • Difficult contract in 2nd half
  • Schedule conflicts

Trending well

Based on feedback, growth metrics, and observations, we see yAcademy scaling, and trending in the positive direction. We continue to encourage the learning, teaching, sharing of web3 security knowledge, and collaboration across open-source. We are also proud to have met, and brought together so many enthusiastic and intellectually-curious minds across the globe who remain kind, open, and respectful. At the end of the day, we are just engineers & scientists who wish to grow a safer ecosystem.

Improvement

As mentioned in Communication and Coordination, our team has a heavy technical bias. Though we continue learning and iterating, the non-technical aspects required for yAcademy to succeed are still under heavy duress. Operations, communication, coordination, will continue to be a focal point as we begin traveling the road of self-sustainability.

We will also continue experimenting with parameters such as interview process, block format, and onboarding resources, to enhance fellowship experience.

What’s next

We are heartened by the ever increasing number of applicants to our fellowship programs, and will begin preparation for the next one in the coming weeks. We ask those who applied to be patient as we ramp up our internal ops/comms/process to effectively deal with the higher load of applicants, onboarded residents, and the audit requests we receive.

Meanwhile, our core and resident auditors continue to receive and take on audit requests from various projects, many of which are returning clients. That work is focused and is not affected by the admin work to organize and run the fellowships. Stay tuned for updates!

--

--

yAcademy
yAcademy

Published in yAcademy

There is an acute shortage of software auditing talent. Considering how much value is at stake in blockchain systems, solving this bottleneck is essential to getting these trust-minimized systems to the a wider audience.

Responses (1)