Intelligent benchmark with wrk

wrk is a modern HTTP benchmarking tool capable of generating significant load when run on a single multi-core CPU.

you can find the project here https://github.com/wg/wrk.

We will install it and do some stupid benchmark, and then we will change dynamically some tests.

Install

Mac

brew install wrk

On Ubuntu

sudo apt-get install build-essential libssl-dev git -y
git clone https://github.com/wg/wrk.git wrk
cd wrk
sudo make
# move the executable to somewhere in your PATH, ex:
sudo cp wrk /usr/local/bin

Docker

You also can use wrk on a docker container : https://hub.docker.com/r/williamyeh/wrk/

Benchmark an HTTP endpoint

wrk -t12 -c400 -d30s --latency http://127.0.0.1:8080/index.html

This runs a benchmark for 30 seconds, using 12 threads, and keeping 400 HTTP connections open.

Output:

Running 30s test @ http://localhost:8080/index.html
  12 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   635.91us    0.89ms  12.92ms   93.69%
    Req/Sec    56.20k     8.07k   62.00k    86.54%
Latency Distribution
  50% 250.00us
  75% 491.00us
  90% 700.00us
  99% 5.80ms  
22464657 requests in 30.00s, 17.76GB read
Requests/sec: 748868.53
Transfer/sec:    606.33MB

You can see the average latency of the request, but most important (with the parameter --latency) the quantile.

Intelligent Benchmark

But what if we want to do more intelligent benchmark, and be able to script to change some parameters ???

it’s also possible with the parameter -s to send lua script because wrk supports executing a LuaJIT script , and that’s fantastic, to be able to script our benchmark tests.

Before explaining how it works, lets first do an Example.(Because an example talks more than 1000 words)

Example :

We will with a very small script, change at each request, the path of the request :

-- init random
math.randomseed(os.time())
-- the request function that will run at each request
request = function() 
   
   -- define the path that will search for q=%v 9%v being a random 
      number between 0 and 1000)
   url_path = "/somepath/search?q=" .. math.random(0,1000)
-- if we want to print the path generated
   --print(url_path)
-- Return the request object with the current URL path
   return wrk.format("GET", url_path)
end

thanks to this script we will be able to change the path of the request randomly at each request :

wrk -c1 -t1 -d5s -s ./my-script.lua --latency http://localhost:8000

others example can be found here : https://github.com/wg/wrk/tree/master/scripts

Wrk and Lua (what can we do ?)

wrk plug with lua 3 useful functions, that will help lua to interact with wrk.

function wrk.format(method, path, headers, body) 

wrk.format returns a HTTP request string containing the passed parameters merged with values from the wrk table.

function wrk.lookup(host, service) 

wrk.lookup returns a table containing all known addresses for the host and service pair. This corresponds to the POSIX getaddrinfo() function.

function wrk.connect(addr) 

wrk.connect returns true if the address can be connected to, otherwise it returns false. The address must be one returned from wrk.lookup().

wrk run in 3 phase :

• setup
• running
• done

Setup

The setup phase begins after the target IP address has been resolved and all threads have been initialized but not yet started.

At this point you can change for each thread being execute the addresses found during the Ip target phase. you are able to manipulate

• thread.addr : get or set the thread’s server address
• thread:get(name) : get the value of a global in the thread’s env
• thread:set(name, value) — set the value of a global in the thread’s env
• thread:stop() — stop the thread

You just have to define your setup function on your LUA script

setup = function(thread)   
   -- code
end

example : https://github.com/wg/wrk/blob/master/scripts/addr.lua

Running

The most useful one.

Running phase are split in 3 others steps :

• init
• request
• response

these 3 steps are 3 of the 4 functions define in LUA to interact with wrk

function init(args) 

The init() function receives any extra command line arguments for the script which must be separated from wrk arguments with “ --“.

function delay()

returns the number of milliseconds to delay sending the next request.

function request()

returns a string containing the HTTP request. Building a new request each time is expensive, when testing a high performance server one solution is to pre-generate all requests in init() and do a quick lookup in request().

function response(status, headers, body)

is called with the HTTP response status, headers, and body. Parsing the headers and body is expensive, so if the response global is nil after the call to init() wrk will ignore the headers and body.

At these points we can interact (thanks to the wrk.format function and the wrk object) :

wrk = { 
   scheme = “http”, 
   host = “localhost”, 
   port = nil,
   method = “GET”,
   path = “/”,
   headers = {},
   body = nil,
   thread = <userdata>,
 }

Done

function done(summary, latency, requests)

The done() function receives a table containing result data, and two statistics objects representing the per-request latency and per-thread request rate.

At this point you can compute your own metrics, the way you want.

Duration and latency are microsecond values and rate is measured in requests per second.

On defining the done function on your lua script :

done = function(summary, latency, requests)   
     -- code
end

example : https://github.com/wg/wrk/blob/master/scripts/report.lua

you can manipulate :

• latency.min : minimum value seen
• latency.max: maximum value seen
• latency.mean : average value seen
• latency.stdev : standard deviation
• latency:percentile(99.0) : 99th percentile value
• latency(i) : raw value and count

summary = {    
    duration = N,  -- run duration in microseconds    
    requests = N,  -- total completed requests    
    bytes    = N,  -- total bytes received    
    errors   = {      
       connect = N, -- total socket connection errors      
       read    = N, -- total socket read errors      
       write   = N, -- total socket write errors      
       status  = N, -- total HTTP status codes > 399      
       timeout = N  -- total request timeouts    
   }  
}

Annexe

Install lua on linux :

curl -R -O http://www.lua.org/ftp/lua-5.3.4.tar.gz
tar zxf lua-5.3.4.tar.gz
cd lua-5.3.4
sudo apt-get install libreadline-dev
make linux test
sudo mv src/lua /usr/local/bin/
sudo mv src/luac /usr/local/bin/

Useful Link to go further :

https://github.com/wg/wrk.

wg/wrk

https://github.com/wg/wrk/blob/master/scripts/

How To Benchmark HTTP Latency with wrk on Ubuntu 14.04 | DigitalOcean

https://hub.docker.com/r/williamyeh/wrk/