How to Organize Judging for Your Virtual Hackathon
Judging has always been my least part of the hackathon organizing experience. I’ve never been to a hackathon that ran a straight forward and orderly judging process. My hope for this tutorial is to help hackers run their judging process smoothly for their virtual hackathons this year.
By the end of this tutorial, you will learn one way to run judging at a virtual hackathon.
Before the Event
Step 1
Establish the judging criteria. At TechTogether, we judge hackers on criteria like the originality, execution, and usefulness of their project. Here is an example of the full judging criteria from HobbyHacks, TechTogether’s first virtual hackathon.
Step 2
Create a submission form that judges will use to evaluate each project. Each project will be evaluated on a scale of 0–10 for each criteria category. I would recommend collecting the following information for each submission: the number of the project they are judging, the challenges they are judging the project for, rating questions for each criterion, and an area for general project feedback.
Here is an example of a judge submission form from HobbyHacks.
Step 3
Create judging instructions that walk judges through the entire process. The instructions should be short, simple, and straightforward.
Here is an example of the judging instructions from HobbyHacks. I adjusted the document, so it could be shared publicly.
Step 4
Create a judging handbook with quick links to the materials you just created. Here is an example of the judge handbook from HobbyHacks.
Step 5
Schedule an onboarding session for your judges right before judging begins. Account for 30 minutes for reading through the judging instructions and answering any questions.
Step 6
If a company is sponsoring a challenge wants to select the winner, make sure to ask them how they want to proceed with judging. Ask them the following questions:
- Will you be providing your own judges?
- Do you want to use our submission form or use your own form of judging?
Example: A sponsors want to look at all the projects submitted to their challenge and pick a winner, separate from your general judging process.
Make sure that all the other judges know not to evaluate projects for the sponsored challenges.
Judges from sponsoring companies should still attend the onboarding session to get the list of projects they need to judge.
During the Event
Step 7
Ok, so hackers just submitted their projects and now you need to prepare for judging. First, you need to turn on the “Gallery” by navigating to “Manage Hackathons” and then “Submissions”. You will need Member access to complete this step.
Step 8
Export all of your project submissions from Devpost by navigating to the “Metrics” tab. Upload it to a spreadsheet in Google Drive and share it with your judges.
Step 9
Create two columns in the spreadsheet titled “Judge 1” and “Judge 2”. This is where we will assign the judges to projects.
Create another column and number each of the projects. You can achieve this quickly in excel by clicking the right corner of the first box in the column and then dragging down.
Step 10
Command “F” for projects that submitted to sponsor-specific challenges and put them in a separate sheet in the same google spreadsheet. Do not remove them from the main sheet if they applied to any other challenges. Sponsors will use this sheet to figure out what project submissions they need to judge.
Assign judges to projects in the columns you just created in the main spreadsheet. We had about 55 projects at HobbyHacks and roughly 20 judges, so we assigned each judge ~7 projects. Each project was judged twice in an attempt to minimize bias.
Step 11
Now, you’re reading for your onboarding session! Go over the judging instructions you created and answer any questions they may have. I recommend budgeting 90 minutes for hackathons with 50 project submissions, which includes the onboarding session.
Calculating the Winners
Step 12
After judging ends, you will need 30 minutes to calculate the winners, prep for the closing ceremony, and check for cheating. Assign one person to handle each of these three tasks.
Step 13
If sponsors selected winners on their own, have them email you the name and number of the winning project.
To calculate the winners, export the data from your judge submission form. First, add up the submission scores sent in by the judges through the form. Once you have those totals, average the two scores each project was awarded.
The teams with the highest scores for each challenge are selected as the winners. If there is a tie, your team can decide the winner. TechTogether has a rule that no team can win more than one prize, but your organization doesn’t have to follow that rule.
Step 14
Pass off the names of the winning teams to your teammate prepping for the closing ceremony. While they prepare, check each of the winning projects for cheating. Each hackathon has different rules on cheating, so I won’t instruct you on how to check for cheating.
Step 15
You are all set! Make sure you have a way for hackers to claim their prize and relax. If you’re lucky, you won’t have to organize judging for a hackathon ever again. :)
