Benchmarking in .NET: Implementation with Intermediate Features

In the previous article, we discussed the literal and technical aspects of benchmarking and its significance and then we went through a simple demo of how to implement benchmarking in .NET.

Dive deep into the ocean of knowledge, where every wave is a new insight waiting to be discovered.

Keeping this in mind, in this article, we will dive a bit deeper to understand and implement some intermediate features that benchmarkdotNet offers for performing benchmarking in .NET.

Let's create a demo project (Console application) for performing the benchmarks.

1. Follow the steps that were provided in the previous article to create a console application and install BenchmarkdotNet Nuget package in it.

Created a Console Application named DemoBenchmarkIntermediate

2. Remove the existing code in program.cs and paste the below code in it

using System.Reflection;
using BenchmarkDotNet.Running;

BenchmarkSwitcher
    .FromAssembly(Assembly.GetExecutingAssembly())
    .Run(args);

If we have more types and we have to choose from which particular benchmarks to run, either by console line arguments or console input, we should use BenchmarkSwitcher.

3. Let's create our benchmark class in program.cs.

public class MyBenchmarks
{
    [Benchmark]
    public void MyBenchmark()
    {

    }
}

This needs to be a public class that should be non-static and non-sealed and that's because something about how the BenchmarkRunner wants this class to be.

Note: Make sure you provide public non-sealed non-static types with the public [Benchmark] method.

4. Our Benchmark won't do anything right now as our benchmark method doesn't have any task to perform but let's run the console app and see how it works.

Note: Build the project in Release mode and run the project with “Start without debugging”

As we are using BechmarkSwitcher, it will first display how many benchmarks, that we have coded, are available under Available Benchmarks. Then it will ask us to select a single or multiple target benchmarks by typing the benchmark number or benchmark caption in the console window.

As we provided 0 as a target benchmark input on the console and pressed enter, it started performing benchmarks on method MyBenchmark() of the MyBenchmarks class.

4. Have you noticed something while executing the benchmark in the previous step? it took quite a lot of time to perform benchmarking on a method that doesn't even contain any statements/operations in it.

The reason is that it has done a lot of warm-up/iterations to perform benchmarks on the specified class. In this particular case, when we have little or no operations inside our benchmark method, we can consider an attribute [ShortRunJob] that will be going to do a lot less prior work.

Moreover, if we consider a case in which we have quite a lot of stuff inside our benchmark methods then we should consider using [MediumRunJob], [LongRunJob] or [VeryLongRunJob] attributes on our benchmark classes.

Let's add something to our Benchmark method.

You can remove the [ShortRunJob] attribute from the benchmark class as we are about to add some work inside our benchmark method.

[Benchmark]
public void MyBenchmark()
{
        List<int> list = new List<int>();
        for(int i=1; i<=1000; i++)
        {
            list.Add(i);
        }

        list.Sort();
}

Based on the given code, we’ve made a list of integers. We filled it with numbers and used the built-in sort function to arrange the numbers in order.

For getting memory consumption-related columns in our benchmark results, we can put [MemoryDiagnoser] attribute on our benchmark class as shown in the below screenshot.

If our goal is to benchmark the list sorting process, it’s unnecessary to include the list creation and number filling in the benchmark method. We should separate this part.

Let's keep this part in a new method separate from the benchmark method.

To make sure that this piece of code should run before the benchmark method, we could use the [GlobalSetup] attribute on this new method. This means that this method will be executed before all other benchmark methods.

[MemoryDiagnoser]
public class MyBenchmarks
{
    List<int>? list;

    [GlobalSetup]
    public void InitialSetup()
    {
        list = new List<int>();
        for (int i = 1; i <= 1000; i++)
        {
            list.Add(i);
        }
    }

    [Benchmark]
    public void MyBenchmark()
    {
        list!.Sort();
    }
}

Note: The “?” sign indicates to the compiler that the list can be nullable, while the “!” sign ensures that when calling .Sort(), the list will not be null at that moment.

Now, instead of running benchmarks only with a list size of 1000 or one by one for various list sizes, if we want to run benchmarks for different ranges of list sizes simultaneously and compare their results, BenchmarkDotNet has a feature that can help us with this!

In the below code and screenshot, we have specified a public field ListSize of integer type and a [Params()] attribute on it that contains a range of different list sizes on which we want to perform our benchmark.

[MemoryDiagnoser]
public class MyBenchmarks
{
    List<int>? list;

    [Params(1000, 10_000, 100_000)]
    public int ListSize;

    [GlobalSetup]
    public void InitialSetup()
    {
        list = new List<int>();
        for (int i = 1; i <= ListSize; i++)
        {
            list.Add(i);
        }
    }

    [Benchmark]
    public void MyBenchmark()
    {
        list!.Sort();
    }
}

Its showtime! Let's run the benchmark and see the results.

Note: Build the console application in Release mode and run the solution with the Start Without Debugging option

Select the target benchmark by typing 0 or a contained benchmark caption MyBenchmarks

Finally! we have successfully executed the benchmark on our sorting list operation and below are the results:

Based on the provided screenshot, the results indicate that sorting a larger list requires more time and memory compared to sorting smaller ones.

In summary, we’ve explored some advanced features that BenchmarkDotNet provides to make benchmark testing more effective.

Happy winters with happy coding :)