Tutorial: getting started with code coverage on your GitHub repositories

This tutorial covers a step-by-step guide to integrate Codecov into your GitHub repositories and generate reports through CircleCI.

The tutorial is aimed at beginners, to cover the basics and to get them started using code coverage.

Example of a coverage report in Codecov.

Table of contents:

1. What is code coverage?
2. Code coverage tools.
3. Why this tutorial focuses on Codecov.
4. Before we begin, set up your repository and CI-solution.
5. Setting up Codecov
6. Reading your first coverage report.
7. Easter eggs / additional content.
8. Limitations of CC, an example.

1. What is code coverage?

Code coverage is a metric to measure how many lines or how large a fraction of the source code is executed by the automated tests. This means that if a program has 100% test coverage, every uncommented line of code is run by the tests.

There are several different coverage criteria’s that can be included, e.g. branch coverage, which focuses on which branches (such as if, else, case-statements) that are covered by the tests.

100% code coverage does not mean that the code is free of bugs, just that the tests execute every line at least once. While code coverage does not measure the quality of your code, it can be used to encourage good programming practice, i.e. to test your code. This will be shown with an example later in the tutorial once you’ve learned a bit more about code coverage.

2. Code coverage tools

Code coverage tools can be used to automatically review code pushed to the repository. Many solutions support rulesets to be created, such as a certain code coverage threshold that must be met for the code review to pass. Many tools offer web-services where the coverage can be tracked. Usually, the code coverage tool receives the coverage report from the test suite and creates a visual presentation of the results.

There are many different flavors when choosing tools to monitor code coverage. The tools covered in this tutorial, all supports multiple programming languages and integration with GitHub repositories. They also offer hosting services to get you started.

Bear in mind that pros and cons are subjective and that this comparison is made to bring some basic understanding of some of the alternatives. Please do explore the tools on your own to find out which one is right for you. There are of course many great alternatives to the tools mentioned in this tutorial.

Coveralls

Coveralls is a web service to track your code coverage over time. The free version of Coveralls (Coveralls Cloud) has the prerequisite that your repository is hosted on GitHub, BitBucket, or GitLab.

Pros:

• Support for most common programming languages.
• Free to use on open source projects.
• Github App support.
• Coveralls has been around for a long time which means that the software has had time to mature.
• The tool is used by many large companies such as Google, Docker, Nvidia to name a few.

Cons:

• The GUI is not very intuitive.
• Not as frequently updated when compared to other alternatives.

Codecov

Much like Coveralls, Codecov allows you to track your code coverage over time via their web-service. Your code needs to be hosted on GitHub, Bitbucket or GitLab.

Pros:

• Free to use on open source projects and small private projects with up to 5 users.
• Support for most common programming languages and then some.
• Can be used together with any CI service.
• Github App support.
• Open-source.
• Used by large companies, such as Facebook, Spotify, Twitter.
• Frequently updated.

Cons:

• No obvious cons, does not mean that none exists.

Code Climate

Code Climate Quality offers automated code review for test coverage and maintainability. You can easily follow your coverage reports via the web service.

Pros:

• Support for common programming languages.
• Free to use on open source projects.
• Github App support.
• Checks for additional problems in the code such as complexity and duplication.
• Ability to run analysis locally with IDE/editor integrations.

Cons:

• Compared to Codecov and Coveralls, Code Climate supports fewer programming languages.

3. Why this tutorial focuses on Codecov

In this tutorial, I chose to use Codecov for the step-by-step guide. The reason why is because it supports more programming languages than the other alternatives and that it is frequently updated. I also find the GUI of the web-service user friendly and like the fact that the tool is free to use for small private repositories.

I would recommend any of the alternatives presented in the comparison as it is a matter of taste.

4. Before we begin, set up your repository and CI-solution

You can skip this part if you are comfortable setting up a test-suite and CI-service for your project.

Set up Maven and generate reports

As an example, we will set up a Java Maven-project with CircleCi. Here, you can learn more about how to create a basic Maven-project with JUnit.

In short, you can create your Maven-project by entering the following line in the terminal:

mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false

This will create a project called my-app in your current directory.

We also want Maven to generate a coverage report that we can upload to Codecov. We can use Jacoco for this by including the following in the plugins section in the pom.xml:

<plugins>
...
  
     <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.8.3</version>
        <executions>
          <execution>
            <id>prepare-agent</id>
            <goals>
              <goal>prepare-agent</goal>
            </goals>
          </execution>
          <execution>
            <id>report</id>
            <phase>prepare-package</phase>
            <goals>
              <goal>report</goal>
            </goals>
          </execution>
          <execution>
            <id>post-unit-test</id>
           <phase>test</phase>
            <goals>
              <goal>report</goal>
            </goals>
            <configuration>
              <!-- Sets the path to the file which contains the execution data. -->
              <dataFile>target/jacoco.exec</dataFile>
              <!-- Sets the output directory for the code coverage report. -->
              <outputDirectory>target/my-reports</outputDirectory>
            </configuration>
          </execution>
        </executions>
        <configuration>
          <systemPropertyVariables>
            <jacoco-agent.destfile>target/jacoco.exec</jacoco-agent.destfile>
          </systemPropertyVariables>
        </configuration>
      </plugin>
...
</plugins>

This will generate a code coverage report located at target/my-reports/jacoco.xml. This is the report that we will upload to Codecov.

Setting up CircleCI

Now that you have a build and test tool running on your repository, you are ready to integrate it with CircleCI.

1. Authorize CircleCI to your GitHub account.

Since CircleCI is available as a GitHub App, you can easily set it up using GitHub marketplace.

CircleCI GitHub App installation.

2. Configuring CircleCI

After installation, you will be redirected to the “Projects” page of CircleCI. You can now choose for which project you would like to set up CircleCI. Simply press the button “Set Up Project”.

The next step is to select a config template. Since we made a Maven-Project, we choose Maven in the list. The content of the template is shown in the black window and can be edited if needed. To keep things explicit, we will edit the config template. Our new config template should look like this:

CircleCI config editor.

Here, you have the same content which you can copy-paste:

jobs:
  build:
    docker:
      - image: circleci/openjdk:11-jdk
    steps:
      - checkout
      - run: mvn test
      - store_test_results:
          path: target/surefire-reports

This will be the content of the config.yml which is the configuration file for CircleCI. Simply put, the config.yml-file tells CircleCI how to build and test your software. Here, you can read more about configuring CircleCI.

Press “Start Building” to proceed.

You will now be prompted if you want to add the config.yml to be added automatically or if you prefer to add it manually.

Press “Add Config” to have the config.yml added to your repository automatically on a new branch.

As soon as config.yml is added to the project, CircleCI will build and run the tests. You can now go visit your repository on GitHub to check the commit status. If all checks pass, you can merge to your master branch.

If your build finishes with errors, check that you have set up your build tool and config.yml properly. You can also press “Details” on the commit status to check the log from CircleCI.

5. Setting up Codecov

As we are finished setting up a GitHub repository with CircleCI, we are now ready to integrate Codecov into our repository.

Authorize Codecov on your GitHub repository.

As with CircleCI, Codecov is available as a GitHub App, link here. You can either choose the free or full-featured version. In this tutorial, we will cover the free version which can be used for open-source projects. Press install to proceed.

Installation of Codecov GitHub App.

You will now be prompted to select which repositories Codecov should be granted access. Choose whatever option you like but make sure to include the repository you are using when following this tutorial. You can also edit the permissions later. Press install to proceed.

Now Codecov is given permission to read and write to your repositories. You should be redirected to codecov.io and be all set to add coverage for your repositories.

Integrate Codecov with your project.

Hopefully, you got redirected and see your repositories listed on codecov.io and select the one for which you want to add code coverage. Otherwise, you can simply enter the URL manually: codecov.io/gh/<GITHUB_USERNAME>/<REPO_NAME>

Manually entering the URL to your repository is actually recommended by Codecov as not always every repository is correctly listed.

You should now see two steps:

STEP 1- COPY TOKEN

We can actually skip step 1 as we are using CircleCI. If you are using another CI tool you might have check if you need to use an upload token. You can read more about the bash uploader and how to use the token here.

STEP 2- UPLOAD REPORTS

STEP 2 contains the command we will use to upload the report to Codecov. Since we want to integrate Codecov to our CI solution, we want the CI server to run the command. How to run commands depends on the CI solution you’ve chosen, just make sure that you run it after the tests are done so that the report has been created.

Integrate Codecov with CircleCI

All we have to do is to edit our CircleCI config.yml to run the command by adding one more step to the job:

- run:
    name: Upload to CodeCov
    command: bash <(curl -s https://codecov.io/bash)

To clarify, our config.yml should now look like this:

CircleCI config file, now with Codecov integration.

Now we should be all set! The report from Jacoco will automatically be detected and uploaded to Codecov.

You have now fulfilled the tutorial! There is however some additional content that could be helpful for newcomers.

6. Reading your first coverage report

Since our example is really simplistic, the coverage report will be very small but should be sufficient to cover the basics.

Navigate to your project on the Codecov webpage and go to the overview section.

If you scroll down on the page, you will see the files of your repository. If you followed the instructions for creating a Maven project, just one file will be listed, App.java. You can also see that the coverage is 0.0%, which is ok for now since we haven’t added any tests. Press the link of the file to see the coverage report.

When viewing App.java, you will see the uncovered lines of code marked in red. If they were covered in the tests they would have been green.

To the right, you might notice a pink percentage, this is the Codecov Delta. The three fields of the delta, from left to right, can be described as:

1. The absolute scope. This is the current coverage of the whole project.
2. The relative scope. This is the coverage of the lines adjusted in the commit, i.e. the diff coverage. Displayed as ∅ here as this is our first coverage report and therefore there is no change.
3. The change in coverage. The change of coverage compared to the commit’s parent. This field is also displayed as ∅ since this is our first report.

You can read more about the Codecov Delta here.

Codecov delta.

You have now read your first coverage report. Try adding more functionality and tests to your code and follow the progression of the coverage.

7. Easter eggs / additional content

Here, additional content is presented for those who want to learn more about code coverage and using Codecov.

Changing the coverage threshold

Here we will set up a custom Yaml-file to configure Codecov to fit your project. We will see how you can change the coverage range for your project, i.e. the coverage needed for a commit to be accepted by Codecov.

Choose the settings tab from the main project view at Codecov’s webpage and then choose “Yaml” on the left side.

At the bottom, you will now see the following buttons.

To further dive into the possible configuration of Codecov, you might want to press “Read the docs”. Here we will just cover the basics. Press “Create a codecov.yml file” to proceed.

As we only want to change the variables that affect the coverage target of the project, we add the following to the codecov.yml file:

coverage:
  status:
    project:
      default:
        target: 50%
        threshold: 10%

target
Here you define the minimum coverage ratio that the commit must meet to be marked as a success.

threshold
Here you specify the maximum drop in coverage for a commit to be marked as a success.

We now have specified that the coverage of the project must be kept above 50% and that any single commit is allowed to drop the coverage with at most 10%. Those numbers can be changed as your project progress.

There are many more parameters that can be used to customize your coverage reports, you can read more about them here.

Add a badge to your README.md

Have you noticed that many open-source projects show their coverage ratio directly in their readme? This is actually really easy to do and can be done on your projects as well.

Navigate to the settings tab from the main project view at Codecov’s webpage. Then choose “Badge” on the left.

You can actually get your badge in different formats, but as we focus on GitHub in this tutorial, we will use Markdown. Copy the content and paste it to your project’s README.md.

All done! Congratulations on showing off your awesome code coverage.

8. Limitations of CC, an example

Just to make an example where we show the limitations of code coverage, we can look at the following method that swaps the first and last element of a string. See any bugs or signs of bad practice?

We can easily cover all code in one test as long as the string is at least 3 characters long, thus fulfilling the for-loop.

The tests succeed and we decide to push the code. Let’s look at the coverage report.

We now have 100% code coverage of our file. Are we done testing? What happens if the string is empty?

The takeaway here is that even with 100% code coverage, we cannot be sure that all bugs are sorted or that every operation is safe under all conditions.