Using GitHub Actions to Help Review Pull Requests Using Unit Tests and Code Coverage

Published in

Marionete

6 min readOct 27, 2022

GitHub Actions is a tool to automate, customise, and execute your software development workflows directly in the GitHub repository. It allows automating your build, test, and deployment pipelines [1] by creating workflows triggered with push actions to the repository.

In this article, we are going to show how to use the GitHub Actions tool to help review a pull request. We are going to configure a workflow to build and test a Python application every time a push is made to the repository.

Such workflow will run all the unit tests and generate a code coverage measure. When a pull request is open if unit tests fail or the coverage does not meet a minimum threshold, the push becomes invalid, and the code will not be mergeable.

Before diving into GitHub Actions let’s understand some of its basics.

1. GitHub Actions Basics

A GitHub Actions [1] workflow is triggered when an event occurs in your repositories, such as a commit push or a pull request opening. Each workflow contains jobs that can run in sequence or in parallel. Each job runs in a virtual machine called a runner. The jobs have several steps, which can be a script or an action (reusable extensions found in the GitHub Marketplace [2]).

Workflow: a configurable automated process that runs one or more jobs. They are defined in a YAML file in the repository. They can be triggered by events (such as a push), manually or being scheduled.
Job: a set of steps in a workflow that are executed in the same runner. They can be a shell script or an action. In a workflow, jobs run in parallel (default), but dependencies can be added among jobs.
Action: a custom application for the GitHub Actions platform. Usually, they are reusable and perform complex tasks. GitHub Marketplace offers several actions ready to be used.
Runner: a virtual machine that runs the workflows. GitHub provides Linux, Windows and macOS runners.

Once you understand the basics, it’s time to code the workflow.

2. Writing the Python Test Workflow

Now we are going to configure a workflow to build and test a Python application every time code is pushed to the repository. This is done by providing a YAML file in the repository.

2.1. Writing the workflow file

The workflow shown in this article is provided in this repository.

Workflow to build and test a Python application

- The YAML file should be stored in the repository in a folder named .github/workflows/
- In line 3, we set the workflow to be executed every time a push event occurs.
- In lines 5–6, we set an environment variable, which later will be used as a minimum code coverage threshold.
- In line 9, we start setting up our only job in this workflow.
- In line 10, we set up the runner for this job. It will run on an Ubuntu virtual machine.
- In line 12, we start defining the steps to execute.
- In lines 13–17, we set up two actions, the first one will fetch our repository to the virtual machine, and the second one will set up Python 3.7 in the runner.
- In lines 19–22, we have a script to install the required dependencies.
- In lines 24–25, we run a script to generate coverage reports (and run the unit tests as well).
- In lines 27–30, we run a script to read the coverage report and check if the minimum code coverage threshold is met.

If all steps run successfully, the workflow reports back a successful check. If unit tests fail, or the code coverage threshold is not met, it will return a failed check.

Don’t forget to push the file to the repository.

2.2. Configuring the Workflow in Github

After pushing the file, head to the repository Settings page. Under the Branches options, add a rule for the main branch (or any other branch you want the workflow to be applied):

Check the option “Require status checks to pass before merging”
Search for your workflow file in the search bar (it should match the name of the YAML file)
Save changes (button at the end of the page)

And done! The workflow should be configured and triggered to run with every push to the repository. The next step is to test it and see if it works.

3. Pushing code to the repository

Here we are going to test a Python application. Sample source and test codes can be found in this repository.

Create a new branch in the local repository:

git checkout -b feature/new-code

Copy the contents of the src/ and test/ folders. Additionally, copy the .coveragerc and requirements.txt files. Optionally, copy the README.md and .gitignore files.
Commit and push the changes to the remote repository.

git add .
git commit -m 'Add source and test codes'
git push origin feature/new-code

Head to GitHub and open a Pull Request.

Pull Request opened — All checks have passed

In the Pull Request page, you should see a green check ✅ next to the commit hash, which means the workflow executed successfully and the code meets the minimum required coverage.

If you click the green check and then follow the Details hyperlink, you should be able to see the workflow logs. If you inspect the logs, you should see a log message stating the coverage is 86%.

Workflow logs show the code coverage achieved

The code is ready to be merged.

4. Pushing (bad) code to the repository

Now, we are going to push changes in the code that will decrease the code coverage. Once the coverage is lower than the previously specified threshold (in this case 75% — section 2.1.), the workflow should mark commit pushes with a failed check. In consequence, merging the commit will not be allowed.

As in the last section, create a new branch.
Comment/remove lines 19–37 in tests/test_person.py:

Commented some test functions to lower code coverage

Commit and push the changes.
Open a pull request.

Pull Request Opened — All checks have failed

You should see a red cross ❌ next to the commit hash. You should also see a message stating the checks have failed, and finally, the Merge button will be disabled.

If you check the logs, you should see the code coverage is now 53.3%. That is the reason the check has failed (keep in mind the minimum threshold required was set to 75%).

⚠️ In case you are the admin of the repository, you may see an option to bypass the check and merge the code. Use it at your own risk.

As an exercise, you could also re-add the tests and change the source code by inserting a bug. When such code is pushed to the repository, the check should fail due to not running all the unit tests successfully (and merge will not be allowed as well).

5. Final thoughts

GitHub Actions is a powerful CI/CD platform that allows building and deploying applications directly from GitHub. This article has shown a simple example of how to help keep a repository clean from bugs and with broad code coverage.

Mind there is so much more that can be done, and there are lots of complex actions ready to be used in the Marketplace [2]. It is also possible to write your own actions that can be re-used in other projects.

GitHub Actions provides solutions for both personal and commercial use. For public repositories, it allows the use of GitHub runners if respecting a usage limit [3]. However, it also allows the use of self-hosted runners [4] which provide more control of the hardware, operating system, and software tools used in them.