Rust benchmarking with Criterion on Travis CI 🦀 ⏲️

Rust comes up with baked-in support for testing without any additional setup. This is definitely a major functionality which makes writing and running test really straightforward: add a few annotations and macros in your file, create some functions that check some results through assertions and then, boom, simply run cargo test 🚀:

But hold on, Rust actually provides more than that! Now you want to add some benchmarks to your code? Here we go:

And cargo bench will run your benchmark tests… if you’re using Rust unstable since the feature is behind a test flag. Hopefully, there’s a really neat solution to address this issue which is called Criterion. It’s been inspired by a library of the same name coming from the Haskell world.

Criterion to the rescue!

Not only Criterion allows you to benchmark against Rust stable, but it’s also providing a set of awesome features:

Statistics: Statistical analysis detects if, and by how much, performance has changed since the last benchmark run.

Charts: Uses gnuplot to generate detailed graphs of benchmark results.

Stable-compatible: Benchmark your code without installing nightly Rust.

I encourage you to have a deep look at its documentation since the aim of this post is definitely not to cover the full potential of this superb library.

Now let’s start scaffolding a simple project that will empower it!

Configure your Cargo.toml file 🛠

The very first step is to start editing your Cargo.toml file to get Criterion in the dev-dependencies and a double-bracket bench section:

Please note that you need to disable the default libtest benchmark harness! You’ll now have to add a benches directory at the root of your project and create a benchmark.rs file inside (the name set in the aforementioned bench section).

Setup your first benchmark 📡

For this next step, we are going to use the Rust 2018 edition. This comes with some major changes, especially regarding the way use declarations are working.

Yup, that’s all you need 🎉.

Let’s quickly have a deeper look at what’s going on here: We have defined a minus_one_benchmark function which takes a &mut Criterion where Criterion is the benchmark manager. The bench_function generates a benchmark with a name — which should be unique — and a closure with one Bencher argument.

There are a couple of macros involved too: a criterion_group! macro, which is grouping our benches (we can inject additional benches as arguments) and a criterion_main! macro which will execute the previous one. Do you remember having disabled the default benchmark harness in the previous step? This macro is taking care of running our benches as expected.

You should now be able to run cargo bench. At this point, I invite you to dive deeper into the documentation.

Define your Travis pipeline 🧪

Now that we have a simple benchmark ready, what about having it running in our TravisCI pipeline, comparing our master branch with the current branch?

The Criterion F.A.Q. provides some guidance on how to reach that goal. However, there are some missing pieces to get it up and running. Let’s go through the necessary steps!

Start by creating your own .travis.yml file at the root of your project. We want to test against Rust stable / beta / nightly. Since nightly can be unstable, the best thing to do is to allow it to fail. We also don’t want to wait for it so let’s use the fast_finish option.

The plan is to set up an additional task to perform the branches’ comparison. We will do it in a shell script called after-success.sh.

You may have noticed an unexpected configuration option here: git depth 10. This is needed in order to avoid getting errors like:

fatal: reference is not a tree: d23e4496e9fbd6d03bac1c9136c32b01af8d3a76

And don’t forget to switch on your pipeline in your TravisCI account!

Add a benchmarks comparison script ⚖️

Congratulations! We’re now almost done! The final step is to create the after-success.sh shell script — by the way, don’t forget to make it executable!

Here we want to clone the repository, bench the master branch (our reference), bench the current branch and compare the two.

Each bench is running with the no-plot option since we don’t want to use gnuplot. Furthermore, we need to save those as named baselines with the --save-baseline flag.

But how to compare those two baselines together? That’s when critcmp comes into play! This command line tool provides an easy way of comparing our benchmarks run by Criterion: critcmp before after.

I’m successfully using this setup for one of my personal project called jql — A JSON Query Language CLI tool. You can have a look at the result of the after-success.sh script e.g. in one of the build here https://travis-ci.org/yamafaktory/jql/jobs/477942582.

Thanks for your time!