Easy C++ Autograding on Github Classroom With Catch2

Autograding tools are very important to improve feedback during programming exercises for students. Hopefully, we already have amazing (and free) tools/platforms to do that! Let’s take a look at Github Classroom for autograding C++ tests using Catch2.

If you want to start learning on practice, just click follow this link, otherwise you can keep reading: https://classroom.github.com/a/X_8PAk4h

First of all, I present the general idea. Teacher creates a template repository for the task, including:

• (i) a construction script (GNU makefile?);
• (ii) a set of tests (written on Catch2 test library?);
• (iii) a README explaining exercises and rules

Each test can receive specific grade, which is validated against pre-defined answers. Student joins an invitation link from github classroom and a private repository is created for its task. Every time it commits, the student grade on the task is recalculated. Teacher is also allowed to interact with students via Pull Request, if necessary.

To start, you need to: (1) create a GitHub Organization — it could be an specific course, or general for all courses you give (2) grant access to GitHub Classroom to your Organization and create a Classroom in the Organization (3) Create an Assignment inside the Classroom. Remember to create a Template repository for the task before creating this, since you’ll be required to add the reference code very soon.

Let’s assume you have an Organization and Classroom (with meaningful names!) and also the Template autograding task, so it’s time to create the assignment:

Giving a name to the autograding task, set as private repository and a deadline

Setup template repository, online IDE, language and execution command

Creating a test case that validates CATCH test named “Ex0” (exercise 0). It receives 10/100 points.

Last test: validates exercise 9 and also the whole pack of exercises, with “make test”

Complete setup for all ten exercises, each receiving grade 10/100. I’m disabling PR feedback for now, since it was not working in a stable manner (in my preliminary testing)

Now you can send the invitation link to the students. As soon as they accept, a copy of template repository will be created for each one of them. Grades are automatically computed here. As soon as activity deadline passes, the invitation link can be removed.

This is how invitation looks like to the student. A nice thing is that you can also subscribe to your own activity and test it!

Ok, it should work! Try to accept your own assignment and push some content there: autograding tests should automatically run and evaluate your repository.

Now, the gotchas…

Important: for some reason, autograding was not running at first, so I discovered that editing the tests would force GitHub to re-setup the autograding. If you have this same experience, change some tests to see if autograding finally works.

Very Important: remember to disable email notifications for GitHub Actions. After few hours for a basic activity released to my 50 students, I received over 400 emails. You can disable this option on: https://github.com/settings/notifications

Disable email notifications for GitHub Actions

Very Very Important: I think there’s some sort of bug, maybe it’s with Repl.it, so for the moment I suggest NOT creating autograding activities with IDE (see problems below).

The strangest thing is that my github user seems to do frequent commits on each student repository. And I don’t know why, as these commits are completely empty, just named “Update GitHub Classroom Autograding Workflow”.

Several autograding commits automatically made on each student repo in few hours

Empty commits named “Update GitHub Classroom Autograding Workflow”

What happened is that, after one week (and around 30 students participating), ALL computing quota was consumed from GitHub (so no autograding was working anymore). In my opinion, it was better this way (for GitHub at least) in order to preserve resources. My solution was to create another Classroom and start over, now without ANY IDE, and fewer testing activities (just put two commands this time, not 10…).

Anyway, community is quite nice and responsive at github, so I opened an issue there: https://education.github.community/t/excessive-update-github-classroom-autograding-workflow/62972

You can follow your organization consumption at: https://github.com/organizations/YOUR-ORGANIZATION/settings/billing/

Now I present some example reference C++ code, if you want to practice.

tests.cpp

// Test definitions. Do NOT edit this file!
#define CATCH_CONFIG_MAIN // defines main() automatically
#include "catch.hpp"
#include "student.hpp" // student implementations
// tests for exercise 1
TEST_CASE("Ex1", "[example]")
{
    REQUIRE(add(1,1) == 2);
}
// tests for exercise 2
TEST_CASE("Ex2", "[example]")
{
    REQUIRE(mul(3,3) == 9);
}

student.hpp (reference file for students to edit)

#pragma once // prevents multiple definitions
// Exercise 1: implements function 'add', that adds two numbers
int add(int a, int b)
{
    return 0; // TODO: fix
}
// Exercise 2: implements function 'mul', that multiplies two numbers
int mul(int a, int b)
{
    return 0; // TODO: fix
}

makefile (build script)

all: clean test
appTests: tests.cpp student.hpp
    g++ -fsanitize=address --std=c++17 tests.cpp -o appTests
test: appTests
    ./appTests
clean:
    rm -f appTests

Here’s the invitation to our Open Classroom. Feel free to play a little bit with C/C++ and Catch2: https://classroom.github.com/a/X_8PAk4h

Note: GitHub has flagged my learning organization for some unknown reason.. I’m working to revert that decision. If the above link doesn’t work, try again in a few days, it should be back.

If, for some reason, the autograding doesn’t run for you, try to click “update” on same the link you received to create the task:

Click “update” to restart autograding on GitHub Actions, if not working for you

It is built upon the following template: https://github.com/igormcoelho-learning/autograding-example-cpp-catch

I already got my 100/100 grade on the practice exercise. What about you? ;) Time to practice!!

Good tests and good luck!