Running Custom Workloads with Sysbench

Sysbench is a popular open-source benchmarking tool for Linux that comes bundled with a suite for benchmarking CPU, memory, file I/O, mutex, and even MySQL. Ease of installation/use and the multitude of options that allow tweaking the execution and reporting of results are the main reasons for its popularity. Instructions for installing from binary packages and basic usage are available in the official documentation at https://github.com/akopytov/sysbench#installing-from-binary-packages.

Aside from the out-of-the-box support for various benchmarks, sysbench supports execution of custom workloads via it’s support for executing arbitrary Lua scripts.

The rest of this post is focussed on the customizability that is built into sysbench to allow execution of arbitrary workloads.

Custom workload execution

Sysbench can execute custom workloads by accepting the path to a Lua script on the command line as illustrated below:

$ sysbench [STD_OPTS] [CUSTOM_OPTS] path/to/script.lua [COMMAND]

A list of standard options (STD_OPTS) and their description/usage can be obtained by runningsysbench — help. We will not discuss these in great detail in this post. Instead, let’s focus on the 3 mechanisms offered by sysbench to aid in authoring custom workload definitions.

Built-in commands

Sysbench supports 4 built-in commands, viz. help, prepare, cleanup, and run. Of these the first 3 built-in commands, (i.e. help, prepare, cleanup) are simply names of global Lua functions that are recognized by sysbench; they don’t come with any useful predefined behavior. Implementing support for these is quite straightforward as illustrated below:

$ cat >builtin_prepare.lua <<EOF
function prepare()
  print("**** prepare() was called.")
end
EOF

Let’s test this:

$ sysbench ./builtin_prepare.lua prepare
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
**** prepare() was called.

Similar results may be obtained by replacing prepare with cleanup or help. The only distinction between these 3 built-in commands is the semantics of the command name itself. i.e. If the underlying behavior is to render a message to indicate usage, it would be best implemented as the help built-in command, whereas thecleanup command may seem more appropriate for behavior that clears state left behind after a test execution.

Implementing support for the run built-in command is a bit different from the other 3 commands, since the run command has an associated lifecycle that is illustrated below:

Lifecycle of the run() built-in command, illustrated here with 3 events across 2 threads.

The custom Lua script is required to provide an implementation for the event() function, while the other lifecycle functions are optional. Depending on the standard command-line options that specify the number of events (either directly using the--events option, or indirectly by specifying the --time option) and the number of threads, the event() function may be called multiple number of times across the various threads.

Let’s see the lifecycle of the run() command in action with an example:

$ cat >builtin_run.lua <<EOF
function init(thread_id)
  print("**** init(" .. (thread_id or "main") .. ") called")
end
function thread_init(thread_id)
  print("**** thread_init(" .. thread_id .. ") called: ")
end
function event(thread_id)
 print("**** event(" .. thread_id .. ") called")
end
function thread_done(thread_id)
  print("**** thread_done(" .. thread_id .. ") called")
end
function done(thread_id)
  print("**** done(" .. (thread_id or "main") .. ") called")
end
EOF

Running this script reveals the sequence in which the lifecycle functions are called:

$ sysbench - events=3 - threads=2 ./builtin_run.lua run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
**** init(main) called
Running the test with following options:
Number of threads: 2
Initializing random number generator from current time
Initializing worker threads…
**** thread_init(0) called:
**** thread_init(1) called:
Threads started!
**** event(1) called
**** event(0) called
**** thread_done(0) called
**** event(1) called
**** thread_done(1) called
General statistics:
total time: 0.0002s
total number of events: 3
Latency (ms):
min: 0.00
avg: 0.01
max: 0.03
95th percentile: 0.03
sum: 0.04
Threads fairness:
events (avg/stddev): 1.5000/0.50
execution time (avg/stddev): 0.0000/0.00
**** done(main) called

Note that init() and done() are called by the main thread, while thread_init(), event(), and thread_done() are invoked by individual threads.

Custom options

The implementation of the built-in commands can be supplemented by specifying custom options for these commands. Let’s take a look at an example that specifies 2 custom options, viz. test_data_csv & num_retries:

$ cat >custom_options.lua <<EOF
sysbench.cmdline.options = {
  test_data_csv = {
    'File to load the test data from.',
    'fixture.csv',
    sysbench.cmdline.ARG_STRING
  },
  num_retries = {
    'Number of retries on failure.',
    3,
    sysbench.cmdline.ARG_INT
  },
}
function help()
  print("Options:")
  sysbench.cmdline.print_test_options()
end
EOF

Note the built-in support for rendering a useful help command. Let’s put this to test:

$ sysbench ./custom_command.lua help
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
Options:
 - num_retries=N Number of retries on failure. [3]
 - test_data_csv=STRING File to load the test data from. [fixture.csv]

The option values can be accessed as illustrated below (following command appends to the Lua script that was created in the earlier step):

$ cat >>custom_options.lua <<EOF
function prepare()
  print("**** test_data_csv: " .. sysbench.opt.test_data_csv)
  print("**** num_retries: " .. sysbench.opt.num_retries)
end
EOF

Let’s test this:

$ sysbench - num_retries=10 ./custom_command.lua prepare
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
**** test_data_csv: fixture.csv
**** num_retries: 10

Note that num_retries gets the overridden value from the command line, where as test_data_csv retains the default value specified in the script.

Custom commands

If the support for built-in commands with custom options is not adequate for any reason, sysbench also provides a way to specify custom commands. Let’s take a look at an example:

$ cat >custom_command.lua <<EOF
function cmd_generate_test_data()
  print("**** cmd_generate_test_data() called.")
end
sysbench.cmdline.commands = {
  generate_test_data = {
    cmd_generate_test_data,
    sysbench.cmdline.PARALLEL_COMMAND
  },
}
EOF

Once again, a quick test shows this in action:

$ sysbench ./custom_command.lua generate_test_data
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
**** cmd_generate_test_data() called.

In Conclusion…

While sysbench is popular for it’s built-in support of various benchmarks, it is also quite customizable through its support for Lua scripting. This post discussed 3 different customization mechanisms: built-in commands, custom options, and custom commands. Complex use cases may find it useful to leverage all of the 3 customization mechanisms.