Continuous integration and code coverage report for a Rust project

When I start a new project, learn a new language, or simply do some coding, I cannot find myself confident if I don’t have a complete continuous integration for it. When using Rust, that’s a must, not because it’s important (it’s always important) but because it’s too easy to setup. There is no excuse.

I will explain how I setup continuous integration in my Rust projects using Travis CI and how to report code coverage to Codecov. Why Codecov? you might ask. Well, I’ve been using Coveralls for some time, but I find their website too faulty. I sometimes don’t see my repos (even if I give them access permissions), sometimes reports don’t work, API fails, and the website itself is designed poorly. Codecov, on the other hand, have a great website, with information presented in a proper way. It works perfectly and has examples for a lot of languages, including Rust. I hope at this time I don’t have to explain why Travis ;)

Continuous Integration

Using Travis CI for continuous integration is easy. Simply sign in with GitHub, and add your repository to your account. Then, you will need to create a .travis.yml file in your repo with the build configuration. And here is where it starts to become interesting. Let’s start with a simple configuration file:

language: rust
cache: cargo
dist: trusty
os: 
  - linux
  - osx
# Run builds for all the supported trains
rust:
  - nightly
  - beta
  - stable
  - 1.0.0
# The main build
script:
  - cargo build
  - cargo test

Lots of things going on here. First, we tell Travis that we want to test Rust code (which installs for us the Rust toolchain in the testing VM). We then add cargo packages to the cache, to speed up new builds, and we select trusty as the testing Ubuntu version (by default, it will use 5 years old Ubuntu, better if we test in 3 years old one). Nevertheless, if testing with older software is required, you can remove that line. Finally, we tell Travis that we want to test both in Linux and MacOS X. This depends on the project, of course (no need to test for MacOS in OpenStratos, it’s only for Raspberry Pi).

Then, we run builds for nightly, beta and stable, and the first Rust version supported by the software. Of course, if our software is only designed to work in nightly, for example, we won’t add stable and beta. I like to add the first supported Rust version so that I can keep track if I break compatibility with old code. Old distributions might not have latest Rust, so it’s best to at least keep track of it. Nevertheless, if I need a new feature, I will break compatibility with older versions, and update the minimum required Rust version to the new feature. Testing on beta enables catching stable-to-beta compiler regressions.

As for the testing scripts, this is up to the project itself. In this simple case, I first build the code, then test it. This differentiates between compilation and unit testing errors. In complex projects, such as OpenStratos, I have builds for each feature set. You can also add packaging tests, for crates being published in crates.io.

Extra features: clippy tests

You can add clippy tests to your nightly CI by adding this before the script tag:

# Add clippy
before_script:
  - |
      if [[ "$TRAVIS_RUST_VERSION" == "nightly" ]]; then
        ( ( cargo install clippy && export CLIPPY=true ) || export CLIPPY=false );
      fi
  - export PATH=$PATH:~/.cargo/bin

This will install clippy only for nightly, and export a variable, CLIPPY if the installation is successful. This can fail because clippy sometimes doesn’t catch up with upstream nightly release breakages. After that, you can test with clippy lints adding this to the script tag:

  - |
      if [[ "$TRAVIS_RUST_VERSION" == "nightly" && $CLIPPY ]]; then
        cargo clippy
      fi

Of course, if you want fine control on those lints, you can add lints to the top of the lib.rs or main.rs files.

Extra features: automatic documentation upload

There is a particular feature that amazes me. With GitHub and Travis, you can upload the documentation of the project to GitHub pages, and make it available for everyone. Of course, now we have docs.rs, but not all projects are in crates.io (for example, many binary projects) or some projects might want developer documentation too.

We will want to upload the documentation only on successful builds (doesn’t make sense to upload docs if the project is broken). For this, we add a new section after the script tag:

# Upload docs
after_success:
  - |
      if [[ "$TRAVIS_OS_NAME" == "linux" && "$TRAVIS_RUST_VERSION" == "stable" && "$TRAVIS_PULL_REQUEST" = "false" && "$TRAVIS_BRANCH" == "master" ]]; then
        cargo doc &&
        echo "<meta http-equiv=refresh content=0;url=os_balloon/index.html>" > target/doc/index.html &&
        git clone https://github.com/davisp/ghp-import.git &&
        ./ghp-import/ghp_import.py -n -p -f -m "Documentation upload" -r https://"$GH_TOKEN"@github.com/"$TRAVIS_REPO_SLUG.git" target/doc &&
        echo "Uploaded documentation"
      fi

This first checks that we are building on linux and on stable (only one upload per build), but can be nightly if the project is not targeting stable. Then checks that it was not a pull request, and that the branch is the master branch (we want the documentation to be only available for the official branch). In that build, we generate the documentation with cargo doc (or with the more complex development documentation command), and we add a small file as the index. This file will redirect visitors to our main crate documentation, in this case os_balloon.

After that, we clone a repository that was created to import github pages easily, generate the commit using our GitHub token. Wait, a token? Yes, we don’t want anybody to be able to push docs (or whatever) to our repos. That’s why GitHub enforces the use of a token. You can generate one here. Make sure you only give it access to the minimum resources needed.

After that, and to protect the token for appearing publicly on build logs, we add it to Travis online configuration. In Travis CI, go to your repo settings, and add the GH_TOKEN environment variable, making sure that “Display value in build log” is off:

GH_TOKEN in Travis CI repository configuration.

After your first successful build in the master branch, you will see the documentation being uploaded to your GitHub pages. You can get the link from the repository configuration in GitHub.

Coverage report with Codecov

As promised, I wasn’t going to talk only about the continuous integration. Code coverage is as important as the CI, since even if we do lots of unit testing during our CI builds, we might not know how to check if we are properly testing all our code paths. That’s where code coverage checking starts to make sense.

Luckily Codecov gives us a great interface and integration with GitHub, BitBucket and GitLab, by using Travis CI or even Circle CI. To enable this, we will need to add some parameters to our travis.yml, as shown in the exmaple. First, after the Rust versions, we will need to add:

sudo: true
before_install:
  - sudo apt-get update
addons:
  apt:
    packages:
      - libcurl4-openssl-dev
      - libelf-dev
      - libdw-dev
      - cmake
      - gcc
      - binutils-dev

This will first update all packages in the VM distribution, and then install some packages required to compile kcov, the tool we will use to collect Rust project coverage results. Then, in the after_success tag, we will add this:

# Coverage report
  - |
      if [[ "$TRAVIS_OS_NAME" == "linux" && "$TRAVIS_RUST_VERSION" == "stable" ]]; then
        wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz &&
        tar xzf master.tar.gz &&
        cd kcov-master &&
        mkdir build &&
        cd build &&
        cmake .. &&
        make &&
        sudo make install &&
        cd ../.. &&
        rm -rf kcov-master &&
        for file in target/debug/examplerust-*[^\.d]; do mkdir -p "target/cov/$(basename $file)"; kcov --exclude-pattern=/.cargo,/usr/lib --verify "target/cov/$(basename $file)" "$file"; done &&
        bash <(curl -s https://codecov.io/bash) &&
        echo "Uploaded code coverage"
      fi

This first makes sure it will only generate coverage reports for stable builds in linux (only once per build). But it will generate reports for all branches and pull requests (ideal if you want to enforce some coverage in PRs). It will then download kcov from its repo, decompress it, compile it, and install it. It will then remove the downloaded repo.

Then, in the for, all the magic happens. It will search for all files in the target/debug starting, in this case, with examplerust-, and report coverage on them. The [^\.d] regex is there because Rust generates some .d ended files without coverage information, that break kcov. You will need to change that beginning of files by your project’s stem. In projects with multiple libs, or external tests/ dir, or both binaries and libraries more than one could be generated.

A simple way of knowing what files get generated is to simply cargo test the project in your local computer. Then you will be able to check all generated files in target/debug :

Generated files for OpenStratos.

In the case of openstratos, we see that some of them start with launcher- and others start with os_balloon-. This is because it has a lib.rs and a main.rs file, but it does not have an external tests directory. In this case, we will need to add two for loops. The loop is required because we might generate more than one test for each binary:

for file in target/debug/os_balloon-*[^\.d]; do mkdir -p "target/cov/$(basename $file)"; kcov --exclude-pattern=/.cargo,/usr/lib --verify "target/cov/$(basename $file)" "$file"; done &&
for file in target/debug/launcher-*[^\.d]; do mkdir -p "target/cov/$(basename $file)"; kcov --exclude-pattern=/.cargo,/usr/lib --verify "target/cov/$(basename $file)" "$file"; done &&

The script ends by getting and executing a bash script from Codecov, that directly sends all the data to their services. You can check OpenStratos coverage reports here. I know, 77% (at the time of this writing) is not enough. We are still on heavy development :)

Extra: badges

What would a repository be without badges? don’t want to know/remember. We can add both Travis CI and CodeCov badges to the repository README, or even to crates.io. For that, we will need to get the badge from Travis at the repo build results. We can get it in markdown, which will look pretty well. Codecov badge can be obtained from the repository settings. Here is how it looks in OpenStratos, you can check the source code for the badges.

For crates.io is even easier. Simply add the badges section to the Cargo.toml:

[badges]
travis-ci = { repository = "Razican/vsop87-rs", branch = "master" }
codecov = { repository = "Razican/vsop87-rs", branch = "master", service = "github" }

You will need to change repository and branch. Note that codecov badge is only available on nightly, and will need a couple of months before you can use it in stable cargo. You can nevertheless use cargo +nightly publish.