Performance and Scale test K8s platform using kube-burner

Kube-burner is an orchestration tool for performing scale and performance tests. It provides following features:

• Create, delete and patch Kubernetes resources at scale
• Prometheus metric collection and indexing
• Measurements
• Alerting

Latest binaries for kube-burner

Depending on the use case you wish to benchmark against, we are able to execute specific workloads. The configuration file for kube-burner contains a description of all of its magic. As was already indicated, the flag -c specifies where this configuration file is located. This indicator links to a multi-sectioned file in YAML format.

Within this configuration file, go-template semantics are usable. Notably, all environment variables are provided to this template, allowing us to reference them with the notation {{.ENV_VAR}}.

Indexing

By default, the metrics from the pod and service latency measurements are sent to every indexer set up in the metricsEndpoints list.

For example

metricsEndpoints:
- indexer:
    type: local
    metricsDirectory: {{ .METRICS_FOLDER }}
- indexer:
    type: opensearch
    defaultIndex: kube-burner
    esServers: ["https://opensearch.domain:9200"]
    alias: os-indexer
global:
  measurements:
  - name: podLatency

The measurement podLatency would employ both local and opensearch indexer with the setup snippet above.

Global

global:
  measurements:
  - name: podLatency
  gc: true
  waitWhenFinished: true
  clusterHealth: true

kube-burner connects k8s clusters in following precedence:-

KUBECONFIG environment variable > $HOME/.kube/config >In-cluster config (when kube-burner runs inside a pod)

measurements Support podLatency, serviceLatency and pprof

gcCleanup created namespaces

waitWhenFinishedWait for all pods/jobs (including probes) to be running/completed when all jobs are completed

clusterHealthChecks if all the nodes are in "Ready" state

Jobs

It contains the list of jobs kube-burner will execute

jobs:
  - name: <job-name>
    jobIterations: 100
    qps: 10
    burst: 20
    preLoadImages: false
    namespacedIterations: true
    namespace: <namespace-base-name>
    waitWhenFinished: true
    podWait: false
    objects:
      - objectTemplate: deployment.yml
        replicas: 1
      - objectTemplate: configmap.yaml
        replicas: 1
      - objectTemplate: service.yml
        replicas: 1

nameJob name

jobIterationsHow many times to execute the job

qpsLimit object creation queries per second

burstMaximum burst for throttle

preLoadImagesKube-burner will create a DS before triggering the job to pull all the images of the job

namespacedIterationsWhether to create a namespace per job iteration

namespaceNamespace base name to use

waitWhenFinishedWait for all pods/jobs (including probes) to be running/completed when all job iterations are completed

podWaitWait for all pods/jobs (including probes) to be running/completed before moving forward to the next job iteration

objectsList of objects the job will create. Detailed on the objects section

Objects

The golang template library by default is used to render the objects generated by kube-burner.

objects:
- objectTemplate: deployment.yml
  replicas: 3
  waitOptions:
    kind: Pod
    labelSelector: {kube-burner-label : abcd}
    forCondition: Ready

objectTemplateObject template file path or URL

replicasHow replicas of this object to create per job iteration

waitOptionsCustomize how to wait for object to be ready

kindObject kind to consider for wait

labelSelectorObjects with these labels will be considered for wait

forConditionWait for the object condition with this name to be true

jobConfig.yml

metricsEndpoints:
- indexer:
    type: local
    metricsDirectory: {{ .METRICS_FOLDER }}
- indexer:
    type: opensearch
    defaultIndex: kube-burner
    esServers: ["https://opensearch.domain:9200"]
    alias: os-indexer
global:
  measurements:
  - name: podLatency
  gc: true
  waitWhenFinished: true
  clusterHealth: true
jobs:
  - name: <job-name>
    jobIterations: 100
    qps: 10
    burst: 20
    preLoadImages: false
    namespacedIterations: true
    namespace: <namespace-base-name>
    waitWhenFinished: true
    podWait: false
    objects:
      - objectTemplate: deployment.yml
        replicas: 3
        waitOptions:
          kind: Pod
          labelSelector: {kube-burner-label : abcd}
          forCondition: Ready

Running Kube-burner tests

kube-burner init -c jobConfig.yml