An In-Depth Guide to Load Testing Solution Powered by K6.

This is the second part in a multi-part series. You should read the first part.

‘Ilities’ : In the realm of software design, numerous ‘ilities’ demand your attention in every project. Prioritization becomes essential because users often expect that you address all of these aspects optimistically.

Today, our primary focus will be on addressing the three ‘ilities’ problem statements mentioned above within our Load Test As A Service and discussing our successful solutions for each of them.

Old load test workflow

Background

We employed K6 for load testing, and had a user-friendly internal wrapper named ‘auto_loadtests’ specifically designed for K6. This wrapper streamlines the setup of crucial parameters necessary for running load tests, including virtual user (VU) settings, load strategies, and script file path along with data generation for the tests.

This wrapper was integrated into a job within the Jenkins service pod, which operated within our Kubernetes (k8s) cluster. Each load test was orchestrated within the auto_loadtests code using JavaScript and initiated from the Jenkins UI layer.

**auto_loadtests : Is a project that empowers developers to write load tests as code, leveraging the capabilities of K6 through wrappers.

Above workflow raised the following considerations. Let’s thoroughly explore each of the problem statements individually: -

Resource consumption: Significant resources were allocated for the Load Test setup, irrespective of the actual load requirements. This often resulted in resource underutilization for smaller loads and necessitates manual efforts, such as simulating pod duplication, to handle high loads.

• It operated within a Jenkins pod configured with resource limits of 1.5 CPU units and 6 memory units in a Kubernetes (K8s) cluster. This configuration used to be consistent even when dealt with smaller load scenarios.
• If there was a need for higher load generation, we would duplicate the Jenkins pod with identical specifications. The increased load was then manually triggered by connecting to the duplicated pod.
• From time to time, when requesting pods with certain specifications, the request would fail because there wasn’t enough capacity on the nodes. This situation required provisioning a new node to handle the request.
• When the load on the Jenkins pod used to increase, it had an impact on all the other pods residing on the same Node.

Load generation occurs external to the K8s cluster: This resulted in increased latency and limited users to focusing exclusively on public APIs for conducting load tests.

• Since the load was generated external to the Application Under Test (AUT) cluster, we were unable to assess a service unless its endpoint was publicly accessible. (Private/Internal APIs, are meant to be used by internal systems, services, or applications. They are not exposed to external users or third-party integrations)
• It introduced additional delay and unnecessary network congestion.

Absence of Auditing: Missing traces of 4W:- Who/What/When/Where per execution.

• By Whom were the load tests initiated?
• What was triggered (Target application, volume of load generated)
• When was this triggered?
• Where the initiation of the load test occurred (Target k8s cluster)?

Alerts, Reporting & Metrics: Absence of comprehensive client and server metrics in a unified view, along with standard notifications with 4W for each load test execution.

• With regards to client-side metrics, we possess an HTML report generated by k6. This report becomes accessible solely when explicitly included in the k6 script and is limited to the most recent execution.
• On load simulated using multiple pods, collecting client-side metrics can be challenging, requiring manual intervention.
• There were no built-in live server metrics available to provide a real-time overview of the server’s request status. (We were continuing to rely on Observability dashboards separately.)
• Slack notifications were managed specific to each script.

Before we start discussing our approaches to tackle the mentioned challenges, let’s take a closer look at the in-house frameworks and tools we’ve utilized to resolve different issues and attain comprehensive solutions.

• APITester :- This Django-based tool serves as a centralized user interface for initiating, overseeing, and troubleshooting the execution of various in-house automation frameworks.
• Validator Controller :- A Flask-based interface is employed for overseeing test executions within a Kubernetes (k8s) cluster and triggering a range of notifications.
• LogCollector :- A Flask-based interface that gathers logs and results, then sends them to APITester, which compiles the reports for each execution.
• Cc Clusters: Kubernetes Clusters as Cap Cloud clusters.

Architecture

High-level solution design:

Instructions for performing a load test using the new solution:

Creation of Load Test Jobs: Generate a Load Test job within the APITester UI, specifying the K6 script path according to the version control system (VCS) repository and selecting the K6 executor type.

• The script path will be employed to analyse the script, extracting all environment variables and displaying them on the UI for user input. These inputs will then be utilized in the execution of the load test.
• The K6 Executor type will be employed to automatically adjust the number of Kubernetes (k8s) pods required to simulate the desired throughput.
• We have confirmed a capacity of 2,000 RPM (Revolutions Per Minute) per pod based on the results of our internal proof of concept (POC).

Execution of Load Testing:

Load test execution flow

• Above execution is either via single or multiple pods which is linked through a runId, a unique identifier for each execution passed as an environment variable to the k6 pods.
• The number of pods will be computed based on the desired throughput, and this desired throughput will be evenly allocated across all the pods.
• The progress of execution will be monitored on the APITester result page for each execution and indicating its status as InProgress, Completed, or Aborted.
• After initiating the load test, the status will indicate ‘InProgress’.(Run ID is the reference of triggered load test) and notification will be sent to slack about 4W.

• In case of an abortion before completion, the status will be marked as ‘Aborted’.
• After the load test finishes, the status will be marked as ‘Completed’.
• Kubernetes (K8s) pods will be terminated upon the completion or abortion of their execution via cluster management Api.

Result Collection: After the successful completion of the load test execution, all result files (in HTML and CSV formats) as well as console logs are routed to the logCollector, which subsequently forwards them to APITester.

• After the execution is finished within the pod.
• We generate a CSV output file by utilizing the ‘ — out csv=loadtestresults.csv’ parameter in the k6 run command. In scenarios involving multiple pods, multiple CSV files are sent to APITester via the logCollector component within various k8s pods.
• We create a k6 HTML result file, but this functionality is not enabled by default. It is controlled through the k6 script, and it becomes active only when users include the following snippet.

const reportBundleUrl = "https://raw.githubusercontent.com/benc-uk/k6-reporter/main/dist/bundle.js";
import { textSummary } from "https://jslib.k6.io/k6-summary/0.0.1/index.js";

export function handleSummary(data) {
 return {
 "result.html": generateHtmlReport(data),
 stdout: summarizeText(data),
 };
}

• We gather console logs and upload them to the API tester via the Log Collector.
• Within the APITester, we access stored log files, CSV files, and HTML files from an S3 repository, which are neatly organized in folders named after the runID.
• While loading the result page in apitester, we perform calculations and save consolidated client-side metrics into a MySQL table. This method eliminates the necessity for frequent recalculations of client-side metrics from the CSV file, and the results are then showcased on the result page.
• For every runId displayed on the results page, you have the choice to retrieve console logs and HTML result files that have been stored in S3.

• We’ve incorporated a Grafana dashboard into the project to visualize up-to-the-minute server metrics derived from K8s ingress logs for public APIs in the Result page.
• We include the runId as the user Agent for each HTTP call.
• We utilize K8s Ingress to filter logs based on runId and perform metric calculations using Loki queries.

Sample graph

Conclusion:

Creating an internal load testing solution can be a practical choice for organizations with unique needs, extensive technical knowledge, and the resources to invest in development, upkeep, and assistance. With our huge customer base, ensuring optimal system performance is increasingly vital. Our project’s primary goal was to enhance the developer experience throughout the end-to-end load testing process with Load Test As A Service.

We successfully addressed various challenges, such as simplifying load test job creation, managing load variations on the Application Under Test (AUT) without concerns about client-server bottlenecks, providing a unified view of metrics (both Client & Server-side), enabling one-click downloads for reports and log files, and offering accessible audit details for 4W (What, When, Where, and Who).

Since its implementation, our developer team has embraced the solution, providing valuable feedback for future enhancements. We are committed to keeping you updated on our progress, so please stay tuned for further updates.