How a Udacity Graduate Earns $11k a Month Reviewing Code
After launching the Udacity Nanodegree program, we faced a crisis: How could we possibly code review tens of thousands of student project submissions each month?
A few months into our inaugural Nanodegree cohort of October 2014, we realized we had a problem: Soon we’d be receiving 15,000 student project submissions each month — 500 projects each day!
So, what is a Nanodegree and why does it require all these projects? A Nanodegree is a new kind of credential for jobs in tech, and embodies the philosophy of learn by doing. Each Nanodegree that Udacity offers is comprised of 4–6 real-world projects. A Nanodegree project could be a mobile app, a web app, a data report or even a well-configured virtual machine. Once you’ve built all of the projects and had them successfully reviewed, you unlock your Nanodegree and become a graduate, with a great shot at starting or upgrading a career in tech.
Scaling Project Reviews
Our challenge was to scale project reviews while fulfilling two objectives. The first was that we had to give quality personal and human feedback to all of our students. We’re a teaching company, so this has been embedded in our DNA from day one. The second was that it had to be fast, meaning that if a student submitted their project on a Sunday at 10AM, they would have their project reviewed by noon that very same day. Fast feedback loops are essential for effective learning.
It was clear that we had to build our very own version of “Uber for Project Reviews”. Whenever a student submitted their project, we’d send an alert to an on-demand workforce of amazing programmers, ready to code review at a moment’s notice.
Our theory was that there was a wealth of programming talent, all around the world, who would jump at the chance to help our students succeed. We hoped that this Udacity Project Reviewer role was to be something you could do during your spare time, but given the volume of project submissions, it was clear there was an opportunity for this to be a full-time gig too.
What is a Project Review?
A Project Review consists of two parts:
- Project Evaluation — Does your project actually do what it’s supposed to do? What are the best practices to follow? What future improvements could you make?
- Code Review — A GitHub-style line-by-line code review, organized by type of comment: “Awesome,” “Nitpick” or “Critical”
On average it takes a Project Reviewer about 30 minutes to perform a project review. It varies dramatically depending on the type and complexity of project, though.
Who is a Project Reviewer?
We had some theories about what types of programmers might make for the best Project Reviewers.
- Freelance programmers (typically charging $70–140 an hour)
- Employed programmers at companies (full-time salary)
- Unemployed programmers (wanting full-time employment with flexibility)
- Udacity graduates (some in jobs, some not)
Some on the team thought we should focus on programmers with professional experience. Although they would cost significantly more than other groups, the theory was that their intimate knowledge of concepts would allow them to teach students in a significantly better way than others. Others felt we should focus on Udacity graduates. There was something magical about receiving a project review from someone who had built the exact same project as you, and knew what trials and tribulations you were going through.
We went on a mission to recruit as many of the four groups as possible. Our pitch was simple: You can earn $50 an hour and work as many or as few hours as you’d like. The best part: You can directly influence a learner’s life. You will be empowered to help a student advance their skill set.
After a few weeks of recruiting, we rounded up a group of over 110 programmers from around the world. Each programmer had different motivations for joining the pilot: Some were in it for the extra money, some wanted to polish their code reviewing skills, but the main theme we heard was that they wanted to give back and support the community.
The Pilot
After 6 weeks of testing it was clear what the next steps were
The results spoke for themselves. The Udacity graduates provided not only the highest rated reviews, but also the fastest turnaround time. This group — in many cases with limited programming experience — outperformed software engineers at Microsoft!
It was a clear and easy choice: We would double down on Udacity graduates as our main source of Project Reviewers. I vividly remember the moment we went all-in with graduates, it was by far the most energizing and fun day I’ve ever had at Udacity.
One of the best revelations was hearing that being a Project Reviewer had proven to be a learning experience. Not only did Udacity students benefit, but our Udacity graduates reported that being a Project Reviewer had made them feel more confident in the workplace with the training and practice they had recieved.
A troubling discovery during the pilot was that per-hour pricing just wasn’t going to work. $50 an hour was still a benchmark we wanted to hit, but the pain and logistical challenges of time tracking were overwhelming. We quickly decided to make the switch to per-project pricing. For example: A complex Android project review might take about an hour, so we’ll pay $50. A JavaScript project review might take 30 minutes, so we’ll pay $25. Keeping our pricing simple would help us launch just that little bit sooner.
The Launch
On February 9th 2015, we were ready to go! We had recruited over 150 Udacity Graduates, who were all ready to review projects after going through a stringent training and onboarding process. We had built a real product that had vital checks and balances, such as student star ratings (ala Uber) and permissions (“Junior Project Reviewer”, “Advanced Project Reviewer”). Not to mention all the features we’d built around payments, line-by-line code reviewing, and project evaluation.
Four months of work was suddenly available to our students, and we immediately heard great feedback. Our Project Reviewers were going above and beyond to help our students from day one.
To top off a great launch, we soon had our first 20 minute project review! A student submitted their project and got a line-by-line code review and evaluation within 20 minutes! The student couldn’t help but email us to say how amazing it was to hear back from a human so fast on a project she had worked so hard on.
Not only were our students benefitting, but our Project Reviewers were too. It was very clear from the first week that a substantial number would be able to make a fantastic income by helping our students.
Today
After 3 months, what have we learned?
We now have Project Reviewers in all of our Nanodegrees, and we’ve seen and heard some amazing things from our students.
- Our average project review turnaround time is now 2 hours. 2 hours from a student submitting their project to having an email in their inbox saying it has been reviewed!
- Our top Project Reviewer for the month of March earned over $11,000! Many earned well over $3,000.
- The students’ average rating of a project review is 4.7/5.0 stars.
- We now have hundreds of Project Reviewers, all around the world, and growing!
Here is some of the feedback we’ve received from students:
“The feedback motivated me to learn more, because they are very specific and constructive. I felt like someone really looked and examined my work. I really appreciate this type and level of feedback. Awesome.”
“I always take any critical feedback as an opportunity to learn. What I learned every time from the reviewer’s feedback is no less than what I learned in your courses.”
“I started programming relatively recently and had been a little insecure about my programming abilities. The reviewer spent a good amount of time pointing out what I had done right and what resources I could use to do other things better. I valued the feedback a lot!”
“I think I am getting hooked on coding and look forward to future projects. Thanks again for all the input.”
“I can’t help but smile right now. The praise is very encouraging!”
“Oh my gosh…that was extremely fast! I was stressing about if I got it right or not and you guys made my night!”
“I really really really loved the feedback on my code. I feel this is the most important part of the feedback that I receive from Udacity, because I hardly ever have anyone look at my code and comment on its quality.”
What I’ve been particularly excited about is where the Udacity team is taking this product. Just one example is we recently launched a beta of résumé review. The goal of a Nanodegree is to enable you to start or level up a career in tech, and with this new tool we’ve enabled students to not just improve their code, but to really help them get their foot in the door of a company. Within a day a student can have actionable human feedback on their résumé, which improves their chances of achieving their goals dramatically.
Just like our Project Reviewers and Udacians helped define the initial product, our Nanodegree students are helping to shape the next version. With their feedback, both good and bad, we’ll continue to help provide faster and better feedback on code.
I’d like to give a huge shout-out to the entire team: Janhavi, Surajit, Eric, Sammy, Lukas, Art, Clifton, Johnathan, Warren, Aparna and Barry. I’m still amazed by how it all came together so quickly. It couldn’t have happened without this extraordinarily talented group of people. If you’re interested in working with Udacians like this, you can apply for one of our many open positions!
