Combining k6, Prometheus remote writes and Vector

to add labels to metrics from k6

Tools:

• k6: a really amazing HTTP benchmarking tool by Grafana.
• Prometheus: a monitoring system, which allow you to store metrics
• Grafana Grafana: a dashboard construction tool, which can show you metrics from Prometheus
• Remove writes: Prometheus feature allowing to ‘accept’ metrics instead of scraping. Normally Prometheus wants to scrape exporters, but remote write allows software to write metrics into Prometheus directly.
• Vector: universal tool for manipulating metrics and logs. For this article it allows to accept metrics like Prometheus in remote write mode, update them, and write them back to the ‘real’ Prometheus.

All this help to build a benchmarking suite, which can generate nice dashboards with results and to have real-time visibility into benchmark progress.

The problem

K6 is amazing, but Prometheus remote write is experimental feature and it does not support additional labels. You can’t meaningfully use flock of servers for benchmarking if you can’t distinct between servers data. Vector is a tool which augment k6. As soon as k6 get own label support, this part can be removed.

Scheme

• k6 runs under systemd unit with specific settings (and specific job in js file).
• It sends metrics into vector (thinking it’s a prometheus remote write)
• Vector transform them
• And send to the real Prometheus
• Grafana shows dashboard with benchmark metrics

k6 + systemd

[Service]
Environment=K6_API_ADDR=127.0.1.0:6565
Environment=K6_PROMETHEUS_RW_SERVER_URL=http://127.0.0.1:9090
Environment=K6_PROMETHEUS_RW_TREND_STATS="p(95),p(99),min,max"
ExecStart=k6 run -o experimental-prometheus-rw --insecure-skip-tls-verify /etc/k6/l4.js
Restart=always

1. K6_API is moved to a different IP to help with multiple jobs on the same server.
2. K6_PROMETHEUS_RW_SERVER_URL points to localhost vector, and 9090 is port for the job. Each job gets own port.
3. k6 run -o experimental-prometheus-rw --insecure-skip-tls-verify /etc/k6/l4.js is running benchmark with remote writes, insecure ssl and a job called ‘l4.js’.
4. Restart=always run job in a loop. It’s not strictly necessary, but I prefer ‘stop/start’ approach instead of ‘run for N minutes’.

Example of l4.js

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
    vus: 1000,
    duration: '3600s',
    rps: 20000
};

export default function () {
    let params = {
        headers: [{"Connection": "close"}],
        insecureSkipTLSVerify: true
    };
    http.get('http://312.411.509.404', params);
    sleep(0.05);
}

The content is well described in the k6 documentation, so I skip explanations.

Single target to run all

[Unit]
Description=Wrk2prom
Wants=k6-l4.service k6-sample.service
[Install]
WantedBy=multi-user.target

A single k6.target allows to run multiple jobs simultaneously, and stop them too. Ansible allows to run it from multiple servers.

ansible -f 5  -bm 'systemd' -a 'name="k6.target" state=started' generators

Vector

I’m old school, so it’s toml. Current Vector recommendation is yaml.

[sources.prometheus_remote_write_l4]  # a different name for each job
type = "prometheus_remote_write"
address = "127.0.0.1:9090"  # the job port number

[transforms.add_labels_l4]
type = "remap"
inputs = ["prometheus_remote_write_l4"]
source = '''
  .tags.server = "gen1"
  .tags.name = "l4"
  .tags.rate_limit_per_server = "20000"
  .tags.rate_limit = "100000"
  .tags.vus_per_server = "1000"
  .tags.vus = "5000"
  .tags.duration = "3600s"
  .tags.proto = "http"
  .tags.servers = "5"
  .tags.target = "http://312.411.509.404"
'''

[sinks.prometheus_out_l4]
type = "prometheus_remote_write"
inputs = ["add_labels_l4"]
endpoint = "http://310.333.376.336:9090/prometheus/api/v1/write"
healthcheck = false

The most magical thing is ‘source’ script, which adds labels. In my case it all is generated by Ansible, including server name and match for port number between k6 and Vector. Note that I add ‘aggregated’ labels from the job description from multiple servers (e.g. vus vs vus_per_server).

Finale

I wanted brag by my dashboard, but this is the job I’m been paid for, so, here just the metrics I get, I hope you can imagine the rest.

k6_http_reqs_total{
  duration="3600s",
  expected_response="true",
  method="GET",
  name="sample",
  proto="http",
  rate_limit="1000",
  rate_limit_per_server="200",
  scenario="default",
  server="gen5",
  servers="5",
  status="200",
  target="http://312.411.509.404",
  url="http://312.411.509.404",
  vus="50",
  vus_per_server="10"
}

Conclusion

prometheus_remote_writes is the future of benchmarks.

k6 + prometheus_remote_writes is really cool.

Vector allow to adjust labels.

systemd allow to group multiple jobs and control their execution.

Ansible allow running them in parallel.

Prometheus store data

Grafana do visualization