Benchmarking Dota 2

Benchmarking a game usually means finding the “Benchmark” button somewhere in the options menu and pressing it. Dota 2 does not come with this feature, however. So in order to find a reproducible testing sequence the user has to do that manually.

As of May 2nd 2018, I’ve rewritten parts of the guide to reflect on a new set of methods I’m using right now. Highlights include much more relevant and conscious data and less reliance on the buggy timedemo command.

The Demo Sequence

The perfect benchmark sequence should ideally represent a workload similar to what a player encounters in-game. The problem is, this approach does not necessarily lead to meaningful numbers. While the majority of the time, players might be running around alone, slaying creeps and heroes alike, the worst-case scenario in a game of Dota 2 usually is a large-scale teamfight. This fight pushes the CPU and GPU to the maximum: Massive amounts of hero models, hats, particles and spells flying around and dropping the framerate.

Pewpew

While I formerly used a replay that was partially in the river which put additional pressure on the CPU, the new replay 5990966909 is significantly more challenging on the particle engine which is more of a bottle neck in the Dota of today. The fight is a late-game fight of two power-house teams, Secret and Liquid, fought out in the Dreamleague Season 15 that served as EU DPC qualifier for the 2021 The International.

The Technical Details

I’ve reworked most of my testing methodology but decided to steer away from the fully automated testing and use a somewhat semi-automated benchmark script with manual input to kick off MSI Afterburner logging.

What we still need:

• A benchmark.cfg which specifies the match id and a bit of console output to be run
• A benchmark.bat which launches Dota 2 with specific commands and calls the benchmark.cfg

First, download the replay ingame (it’s on the free Dreamleague Season 15 ticket) or via Mega and put the 5990966909.dem file into the replays folder in:

Steam\steamapps\common\Dota 2 Beta\game\dota\replays

benchmark.cfg

Navigate to the cfg folder in

Steam\steamapps\common\Dota 2 Beta\game\dota\cfg

and create a benchmark.cfg with the content in this pastebin, or simply download it here.

The .cfg file is mostly inspired (and in parts copied!) by aveyo’s excellent scripts and is navigating the client to the correct time in the replay file and provides a countdown when to hit the begin- and end-benchmark buttons of MSI Afterburner.

benchmark.bat

Create a benchmark.bat file wherever it’s most convenient to you. This script will simply execute Dota several times, tell it to load the benchmark.cfg and simply repeat after the testing was done.

The benchmark.bat has the following content:

@echo on
set dota_path=C:\Games\Steam\steamapps\common\dota 2 beta
set renderers=dx11 dx9 vulkan gl
set res_x=1920
set res_y=1080
for %%r in (%renderers%) do (
  start /wait /d "%dota_path%\game\bin\win64" dota2.exe "-%%r" +con_enable 1 -noborder -windowed -x 0 -y 0 -width %res_x% -height %res_y% +exec_async benchmark +demo_quitafterplayback 1
  timeout /t 3
)

And the settings explanation:

• dota_path: Points to your Dota 2 install location. You have to change this.
• renderers: Dota supports different graphic APIs, using several different ones can lead to interesting results. Right now Direct3D11 (-dx11), Direct3D9 (-nod3d9ex), Direct3D9ex (default), OpenGL (-gl) and Vulkan (-vulkan) are supported.
• res_x & res_y: Specifies the resolution used. The script tries to set borderless windowed mode but that seems to be buggy right now. If that’s the case for you, you can simply set Exclusive Fullscreen or (borderless) windowed mode in the settings in-game.

This script launches Dota, executes the benchmark.cfg and shuts Dota down again. The it starts the next test until all renderers are tested. From years of testing, I only ever test DX11 and Vulkan, but perhaps for some crazy reason other APIs work better for you. You should also run this a few times for sample size errors.

Please note, due to this being a .bat script, this won’t work on Linux, but a simple .sh script will provide similar functionality.

Last but not least, download the Vulkan and GL DLC via Steam. They are free, of course, but aren’t preinstalled.

Collecting frametimes

While the old timedemo command that I used in the older version of this guide provides average fps as well as a variance metric, I now switched to manual benchmarking with MSI Afterburner due to the better data collection. Especially 1% lows are an incredibly useful metric and I personally prefer a smooth game over simply high average fps.

What we’ll need:

• MSI Afterburner and Riva Tuner Statistics Server: Link

Install it and then do the following in the settings dialogue:

• In “Monitoring”, check Framerate AVG and 1% and 0.1% lows
• In “Benchmark”, get some hotkeys of your choice going for “Begin Recording” and “End Recording”
• Still in “Benchmark”, make a results file, like Benchmark.txt, in a folder of your choice and navigate to it

MSI Afterburner settings

Verify that RTSS works by simply launching the game. You should have a bunch of information in the top left corner.

Running the benchmark

1. Run the dota_benchguide.bat file
2. Dota will now open
3. When the console output counts you down, start the benchmark by pressing the hotkey you set in MSI Afterburner
4. When the console outputs counts you down again, stop the benchmark by pressing the other hotkey you set in MSI Afterburner
5. Dota will now close
6. If you specified multiple render options in your .bat file, it will loop back to step 2 now until all settings are done (possibly 5 times if you benchmark all 5 different render paths)

Ingame output while running the script

The Results

After every finished run, MSI Afterburner dumps the new data in the specified .txt file. Disregard the console output, it’s mostly used as control values (and simply isn’t as thorough as MSI Afterburner’s numbers).

Typical output

The data in the file is in the same order as you specified in your .bat file. So if you went with DX11, DX9, DX9ex, Vulkan, OpenGL order, then that’s the same order that you’ll see the data for.

What I’d suggest, however, is to run the benchmark a few times at least. Especially 0.1% lows are incredibly variable and when I did my RAM benchmarking, I used the average of 5 runs each to calculate the final numbers.

That’s mostly it. You can tinker with the .bat file and the .cfg files a lot to adjust them to your needs. You could try different in-game settings and see how they affect your fps for example.

Lastly, if you’re a member of the tech-press and use these instructions for your content, feel free to reach out if you want to chat about Dota 2 and it’s quirks and/or use this guide to your hearts content if you give credit.

JJ, out.