Published in

Rustaceans

9 min readMar 4, 2024

Support me by buying an item from my wishlist, visiting my reviews site or buy me a coffee!

Benchmarking Your Rust Code with Criterion: A Comprehensive Guide

In the realm of software development, performance is paramount. As developers, we strive to write code that not only solves problems but does so efficiently and quickly. This is particularly true when working with Rust, a language renowned for its performance characteristics.

Benchmarking, the practice of testing a piece of software to measure its performance, is a crucial aspect of software development. It allows us to understand how our code performs and where it can be optimized. In the world of Rust, one tool stands out for this purpose: Criterion.

Criterion is a powerful benchmarking library in Rust that provides robust and precise insights into your code’s performance. It offers a variety of features, from basic benchmarking to advanced statistical analysis, making it an invaluable tool for any Rust developer keen on optimizing their code.

In this article, we will delve into the world of Rust benchmarking with Criterion. We will explore why benchmarking is important, how to set up Criterion in your Rust project, and how to use it to benchmark your code. We will also look at some of Criterion’s advanced features and how they can help you write faster, more efficient Rust code.

So, whether you’re new to Rust or an experienced developer looking to squeeze every last drop of performance from your code, this guide is for you. Let’s get started on our journey towards better, faster Rust code with Criterion.

Understanding Benchmarking

Benchmarking is a systematic process of comparing the performance of your software to either a standard, such as an industry best practice, or similar software applications. It’s a way of discovering what is the best performance being achieved, whether in a particular company, by a competitor, or by an entirely different industry. This information can then be used to identify gaps in an organization’s processes in order to achieve a competitive advantage.

In the context of Rust, benchmarking is particularly important due to the language’s focus on performance. Rust is designed to provide the power of low-level languages like C++, but with the safety guarantees of high-level languages. This makes it a popular choice for systems programming and other performance-critical applications. However, with this power comes responsibility. As a Rust developer, it’s important to ensure that your code is not only safe but also efficient.

There are several common methodologies for benchmarking software. These include:

Micro-benchmarks: These are small, isolated pieces of code that are designed to test a specific function or operation. They are useful for testing the performance of individual functions or algorithms.
Macro-benchmarks: These are larger, more complex benchmarks that test the performance of an entire system or application. They are useful for testing the overall performance of a software application.
Load testing: This involves testing the system under heavy loads to see how it performs. This can help identify bottlenecks and areas where performance can be improved.
Profiling: This involves collecting detailed data about how a program executes in order to find areas that can be optimized. This can include data about CPU usage, memory usage, and more.

In the next section, we will introduce Criterion, a powerful tool for benchmarking Rust code, and discuss why it is a suitable choice for these tasks.

Introduction to Criterion

Criterion is a powerful benchmarking library in Rust that provides robust and precise insights into your code’s performance. It is designed to be easy to use, statistically rigorous, and practical for developers who want to keep their code running fast.

What is Criterion?

Criterion is a performance measurement library. Unlike the built-in test framework in Rust, which is primarily aimed at unit testing, Criterion is specifically designed for benchmarking Rust code. It provides a way to write benchmarks that measure the performance of your code in a statistically rigorous way.

Why Criterion is suitable for benchmarking Rust code

Criterion is particularly well-suited to benchmarking Rust code for several reasons:

Statistical Rigor: Criterion uses statistical methods to provide accurate and precise measurements, even in the presence of noise and variability in performance data.
Ease of Use: Criterion is designed to be easy to use. It provides a simple API for defining benchmarks, and it automatically handles the details of collecting and analyzing performance data.
Detailed Analysis: Criterion provides detailed analysis of your benchmarks, including both summary statistics and graphical plots of your data. This makes it easy to understand the performance characteristics of your code.

How Criterion compares to other benchmarking tools

While there are many benchmarking tools available, Criterion stands out for its focus on statistical rigor and ease of use. Other tools may provide raw performance data, but Criterion goes a step further by providing detailed statistical analysis of your data. This can help you understand not just how fast your code is, but also how its performance varies under different conditions and how reliable your measurements are.

In the next section, we will guide you through setting up Criterion in your Rust project and creating your first benchmark.

Setting Up Criterion in Your Rust Project

Benchmarking your Rust code with Criterion begins with setting up the library in your project. This involves a few steps, including installing the library, configuring it in your project, and creating a basic benchmark test.

Installing Criterion

Criterion is available on crates.io, the Rust package registry. You can add it to your project by adding the following line to your Cargo.toml file under the [dev-dependencies] section:

[dev-dependencies]
criterion = "0.3"

Then run cargo build to download and compile Criterion.

Configuring Criterion in a Rust Project

To use Criterion, you need to create a new directory in your project’s root directory named benches. This is where all your benchmark tests will reside.

In the benches directory, create a new file for your benchmark test. The name of the file will be the name of your benchmark. For example, you might create a file named my_benchmark.rs.

In my_benchmark.rs, you’ll need to add the following code to set up Criterion:

use criterion::{criterion_group, criterion_main, Criterion};

fn my_benchmark(c: &mut Criterion) {
    // Your benchmark code goes here
}

criterion_group!(benches, my_benchmark);
criterion_main!(benches);

This sets up a benchmark group named benches that includes the my_benchmark function. The criterion_main! macro generates a main function that runs the benchmarks.

Creating a Basic Benchmark Test

Now that Criterion is set up, you can create a basic benchmark test. In the my_benchmark function, you can use the bench_function method to benchmark a specific function. For example:

fn my_benchmark(c: &mut Criterion) {
    c.bench_function("my_function", |b| b.iter(|| my_function()));
}

This will benchmark the my_function function. The b.iter method is used to run the function multiple times and collect performance data.

In the next section, we will delve into how to write more complex benchmark tests with Criterion and how to interpret the results.

Benchmarking with Criterion

Now that we have set up Criterion in our Rust project, it’s time to delve into the process of benchmarking our code. In this section, we will discuss how to write benchmark tests with Criterion, understand the output, and optimize our Rust code based on the results.

Writing Benchmark Tests with Criterion

Writing benchmark tests with Criterion is straightforward. As we saw in the previous section, we use the bench_function method to benchmark a specific function. The function we want to benchmark is passed as a closure to the iter method.

Here’s an example of a simple benchmark test for a function that calculates the factorial of a number:

fn factorial(n: u64) -> u64 {
    (1..=n).product()
}

fn benchmarks(c: &mut Criterion) {
    c.bench_function("factorial 20", |b| b.iter(|| factorial(20)));
}

criterion_group!(benches, benchmarks);
criterion_main!(benches);

In this example, we’re benchmarking the factorial function with an input of 20.

Understanding Criterion’s Output

When you run your benchmarks with cargo bench, Criterion will output detailed information about each benchmark. This includes:

The average time it took to run the function.
The standard deviation, which tells you how much variability there is in the function’s runtime.
The median time and median absolute deviation, which are robust statistics that can give you a better idea of the typical runtime when there are outliers.
The throughput, which tells you how many times the function could theoretically be run per second.

Criterion also generates detailed plots of the distribution of runtimes, which can help you understand the performance characteristics of your code in more detail.

Optimizing Rust Code Based on Criterion’s Results

Once you have benchmarked your code with Criterion, you can use the results to guide your optimization efforts. For example, if you find that a particular function is slower than expected, you can focus on optimizing that function.

Remember, though, that optimization should not come at the expense of readability or correctness. It’s often said in the Rust community: “Make it work, make it right, make it fast, in that order.”

In the next section, we will explore some of the advanced features of Criterion that can help you gain even deeper insights into your code’s performance.

Advanced Criterion Features

Criterion offers a range of advanced features that can help you gain deeper insights into your code’s performance. These features include statistical analysis, comparison of function performance, and regression testing.

Statistical Analysis with Criterion

Criterion uses a variety of statistical methods to provide accurate and precise measurements. It uses the bootstrap method to estimate the confidence intervals of the mean and median, and the Jackknife method to estimate the confidence intervals of the standard deviation. These methods allow Criterion to provide reliable performance measurements even in the presence of variability and noise in the data.

In addition to these basic statistics, Criterion also provides histograms of the measured runtimes. These histograms can help you understand the distribution of your function’s performance and identify any outliers or unusual behavior.

Comparing Function Performance with Criterion

One of the most powerful features of Criterion is its ability to compare the performance of different functions. This can be useful when you’re trying to decide between different implementations of a function, or when you’re trying to understand the performance impact of a change to your code.

To compare two functions, you can use the bench_functions method:

fn benchmarks(c: &mut Criterion) {
    let mut group = c.benchmark_group("My Group");
    group.bench_function("Function 1", |b| b.iter(|| function1()));
    group.bench_function("Function 2", |b| b.iter(|| function2()));
    group.finish();
}

This will run both functions and compare their performance.

Using Criterion for Regression Testing

Criterion can also be used for regression testing. This involves benchmarking your code after each change to ensure that performance has not regressed. Criterion makes this easy by providing a way to save benchmark results and compare them with future runs.

To save benchmark results, you can use the --save-baseline option when running cargo bench. This will save the results in a file named baseline.json. You can then compare future benchmark runs to this baseline by using the --baseline option.

In conclusion, Criterion’s advanced features provide a powerful toolkit for understanding and optimizing the performance of your Rust code. By using these features effectively, you can ensure that your code is not only correct, but also fast.

Conclusion

Benchmarking is a critical aspect of software development, especially when working with a performance-focused language like Rust. It provides insights into how your code performs and where it can be optimized. Criterion, with its statistical rigor and ease of use, is an invaluable tool for any Rust developer keen on optimizing their code.

In this article, we’ve explored the importance of benchmarking and the role Criterion plays in this process. We’ve walked through setting up Criterion in a Rust project, writing benchmark tests, interpreting the results, and using these insights to optimize our code. We’ve also delved into some of Criterion’s advanced features, such as statistical analysis, function performance comparison, and regression testing.

As we continue to develop in Rust, let’s remember the importance of benchmarking. It’s not just about making our code faster — it’s about understanding our code better. It’s about making informed decisions on where to focus our optimization efforts. And most importantly, it’s about ensuring that our applications provide the best possible performance for our users.

So, keep benchmarking, keep optimizing, and keep pushing the boundaries of what’s possible with Rust and Criterion. Happy coding!🦀