Your benchmarks are so wrong!

TL;DR — If you are measuring a relative execution time of 2 pieces of code, make sure to run under the Visual Studio debugger.

Disclaimer: this post is relevant to Visual Studio debugger. I did not check it on other platforms or debuggers.

How many times have you tested and bench-marked some operation in your code? How many

Yesterday i read a post on the internet that attempted to benchmark 2 pieces of code that perform the same task in C#, and compare the 2 solutions for the same problem (comparing execution time complexity) to decide which is faster. The methodology was to create 2 ‘for’ loops and run them a million times each, one after the other in the same program, when each of the loops measuring it’s own execution time beginning to end.

The outcome was unequivocal, the second loop completed twice as faster than the first one! Amazing! “Not so fast buddy” was my initial thought.

I was skeptic, on the face of it the 2 loops looked like they should perform more or less the same (time-wise). I decided to run the same test on my machine.

Results:

Execution time of the 2 loops on my machine was almost the same (with very minor differences).

The findings:

First, i noticed he ran his code without the debugger. He simply pressed “Ctrl+F5” and ran the program outside of the debugger. Well, i ran mine in debug mode. Interesting. Could that be the answer? Do you see the problem already?

The technical explanation:

When you run your code from Visual Studio when the studio’s debugger is attached, the process is optimized to run at a constant priority which is pretty high. Here are the 2 loops on my machine, notice how the processor is stable in the amount of attention it dedicates to my process. (the drop before the end is the end of the program)

Now take a look how the same graph looks like when i run the same program with the debugger detached. It seems as if the process starts slow and then the system increases the attention it dedicates to the process and the execution speed almost doubles.

Amazing, isn’t it?

The conclusion:

Obviously, if the process priority gets higher when executing the second loop, it will seem as if the second loop performed much faster. This will lead to a complete false benchmark result.

Solution:

The best way to compare the 2 loops is by running them under the debugger of Visual Studio. Another way is to insert a dumb loop at the beginning of your program to “burn” the unstable priority time and continue when the CPU gets up to speed and starts executing at constant priority.

If you are angry, mad or just happy about this post and want to share it with me — leave a comment.

Code on,
Shonn Lyga.