Comparing Two Benchmarks in Go 1.18

Today we will look at a tool that allows us to compare benchmarks quickly and easily via the console

Henry Steinhauer
Better Programming
Published in
4 min readMay 18, 2022

--

Gopher lifts weights
The Gopher was drawn by Egon Elbre.

If you’ve done a bit of development with Go, you’ve probably already noticed how easy it is to benchmark a function and check the memory allocation. Golang provides us with tools and commands that make benchmarking very easy. But what should we do after we benchmarked a function?

Sometimes it is necessary or useful to compare the old version with the new one. For example, you’ve written a function, done a benchmark test, and found that it’s slow as hell. You search the internet and find a better solution. You implement it and run your benchmark test again with your new logic. You think it should be faster, but you’re not sure.

How great would it be to just compare your new result with the old one! When I wrote this article, Golang, unfortunately, didn’t have such a feature implemented yet. But don’t worry. Today, I’m going to introduce a way to do just that. Now, I’ll stop talking. Let’s go.

This is the second article of a new series of mine. If you are interested in generics, you may also find the first article interesting: How to write generic helper functions with Go 1.18

We implement two different functions that calculate the nth Fibonacci number from the sequence for better illustration. You can see our first implementation below.

Simple Recursive Implementation

It is a very simple approach. If n is less than 2, n is returned. In the other case, we call our function recursively with n-1 and n-2. In the end, we return the summed result of both.

Simple Iterative Implementation

In our second implementation, we return n directly if it’s less than 2. If not, we loop n times and sum the value of n-1 + n at each iteration.

Recursive vs. Iterative Approach

Now we have two different implementations. One way to check which of them is more performant would be to run benchmark tests with both functions. Then you put both results side by side, compare them, and voila, we have a winner. But how cool would it be to compare them via the console and get exactly the difference between both benchmarks.

A few months ago, I came across a package that does just that. It’s called benchstat, and it is very easy to use. I will show you how we can use it with our two implementations.

Comparison via Benchstat

First, we need to take benchmarks from our first and second implementations and save them to a file.

Recursive Approach

go test -bench="BenchmarkFibonacci" -run=^# -count=5 | tee fibunacci_recursive.txt

Iterative Approach

go test -bench="BenchmarkFibonacci" -run=^# -count=5 | tee fibunacci_iterative.txt

Next, make sure that you have installed benchstat. If not, install it with the following command:

go install golang.org/x/perf/cmd/benchstat@latest

In the end, we just need to call the command from benchstat to compare both benchmarks.

benchstat fibunacci_recursive.txt fibunacci_iterative.txt

As a result, benchstat displays a statistic with different columns. In the first column, you can see the function's name we tested. The second and third columns show the average time difference over the samples with the percentage deviation for both files. And the last column is called delta. This is where benchstat shows the percentage change in performance between the two benchmarks.

Below, you can see our results from comparing both approaches. Our recursive variant takes exponentially more time per operation the higher the value of our argument. While our iterative implementation increases almost evenly. We can see what this means in a direct comparison by looking at the delta. The difference in performance increases rapidly.

Result of the comparison of two benchmarks using benchstat
Result of the comparison of two benchmarks using benchstat

Final Thoughts

Benchmarking can be very useful, but it also generates a bunch of data. Benchstat can help analyze the data and draw the right conclusions. If you have something to mention or questions, it would be great if you leave them in the comments.

See you soon.

P.S. This is the second article of a new series I have planned. Over the next few weeks, I’ll look at various generic helper functions, interesting benchmarks, and useful features worldwide.

--

--

Henry Steinhauer
Henry Steinhauer

Written by Henry Steinhauer

Passionate software developer who enjoys exploring new programming languages, design patterns and frameworks.

Responses (1)