On few last articles I’ve been mentioned test that included in functional testing. Other than that, we as a QA Engineer/SDET also need to know about performance test and the goals of doing this test.
Performance Test & k6 Definition
Based on BlazeMeter, performance test is a tests that check how the system behaves and performs. As I know, performance test is a basic term of all these testing but not limited to load test, stress test, spike test, and others.
All of those tests are examines responsiveness, stability, scalability, reliability, speed, and resource usage of the software and infrastructure. From what I learn, the difference of load test and stress test is from the scenario to handle. Load testing is a type of performance test that checks how systems behavior under expected or projected number of concurrent virtual users who performing few transactions, meanwhile stress test is a type of performance test that checks the upper limits of the system by testing it under extreme loads.
Performance test can be tested using k6; one of the open source tools that free, developer-centric, and extensible to test the reliability and performance of the system. Script on k6 written in JavaScript code and able to running the test on local device or on the k6 clouds.
Performance Test Goals and k6 Metrics
Before starting on the test goals, it is important to determine the system-business goals, so we can tell if the system behaves satisfactorily or not according to the customer’s needs. From the business goals, we also able to set the performance test requirement, goals, and metrics to analyze as the result of performance test.
k6 already provide some built-in metrics that always collected every time we are running the test, such as.
- vus — define current number of active virtual users,
- vus_max — define the maximum possible number of virtual users,
- iterations — the aggregate number of times the VUs execute the JS script, and
- checks — define the rate of successful checks
Beside that, HTTP-specific metrics also provided every time the test make HTTP requests, such as.
- http_req_duration — define total time for the request; how long did the remote server take to process the request and respond, without the initial DNS lookup/connection times,
- http_reqs — define how many total HTTP requests k6 generated
and many more useful metrics that have been provided (can be found in k6 Built-in metrics page!)
Example of test using k6
On this part, we will try to make a simple performance test using k6. For the sample, we use open API PetStore. Before it, make sure you have installed k6 on local device.
Scenario
Let’s say we expect the submit pet API able to handle 10 concurrent users for 1-minute period with maximum duration of HTTP request should be below 500ms. So that we will performing unit performance test which will test 1 specific endpoint, in this case the endpoint is /pet to submit new pet to the store.
Based on the scenario and Swagger documentation above we know that;
- submit pet API using POST method,
- full URL of the request will be https://petstore.swagger.io/v2/pet,
- the body request should contains at least name and the status in JSON format,
other informations that we know are number of VUs in k6 will be 10; since we expect 10 concurrent users doing the transactions, and the request duration is 1 minute.
Code
To starting the JS script, we need to import the k6, http to create the request via API, as well as check to validate the expectation of the request response.
import http from 'k6/http';
import { check } from 'k6';To define the VUs, duration of the request, and threshold for maximum HTTP request duration, we are able to use option constant as follows.
export const options = {
vus: 10,
duration: '60s',
thresholds: {
http_req_duration: ['max<500']
}
};After that, we can define the static payload and the request to run as created below.
const payload = JSON.stringify(
{
name: 'Dog',
status: 'available',
}
);
const headers = { 'Content-Type': 'application/json' };
export default function () {
const res = http.post('https://petstore.swagger.io/v2/pet', payload, { headers });
check(res, {
'status was 200': (r) => r.status == 200,
'pet name is correct': (r) => JSON.parse(r.body)['name'] == JSON.parse(payload)['name']
});
}The code to run the request is scripted inside the default function using http library and validate if the the status code was 200 and the pet name in response is equal to the pet name in the body request (or payload in this case).
If we don’t want to submit static pet name for all the requests, new method to get random pet name is expected to create.
function getRandomPetName() {
const petList = ['Dog', 'Cat', 'Hamster']
const randomIndex = Math.floor(Math.random() * petList.length);
return petList[randomIndex];
}
export default function () {
const payload = JSON.stringify(
{
name: getRandomPetName(),
status: 'available',
}
);
const res = http.post('https://petstore.swagger.io/v2/pet', payload, { headers });
check(res, {
'status was 200': (r) => r.status == 200,
'response name same with payload': (r) => JSON.parse(r.body)['name'] == JSON.parse(payload)['name']
});
}As shown on the script above, getRandomPetName method has been created to return different pet name by random every time its called. The value of name key in payload also changed to call the method, as well as the position was inside default function.
import http from 'k6/http';
import { check } from 'k6';
function getRandomPetName() {
const petList = ['Dog', 'Cat', 'Hamster']
const randomIndex = Math.floor(Math.random() * petList.length);
return petList[randomIndex];
}
export const options = {
vus: 10,
duration: '60s',
thresholds: {
http_req_duration: ['max<500']
}
};
const headers = { 'Content-Type': 'application/json' };
export default function () {
const payload = JSON.stringify(
{
name: getRandomPetName(),
status: 'available',
}
);
const res = http.post('https://petstore.swagger.io/v2/pet', payload, { headers });
check(res, {
'status was 200': (r) => r.status == 200,
'response name same with payload': (r) => JSON.parse(r.body)['name'] == JSON.parse(payload)['name']
});
}Save the JS script as you wanted the name, and run on Terminal/Command Prompt using k6 run <file-name> and the result will shown as follows.
From the result above, we know that;
- with 10 maximum VUs within 1-minute period, 2282 requests has been created to hit the PetShop API,
- all responses code was 200 and all the pets name in response body are correct,
- the threshold for maximum HTTP request duration is reached which is expected to below 500ms but actual result is up to 900ms (997.92ms to be exact)
In conclusion, the submit pet endpoint in PetShop API still able to handle 10 concurrent user in 1 minute requests, but the maximum response time was exceeded (the threshold is 500ms) which about 997.92ms and average 259.02ms.
Key Takeaways (in my opinion!)
As we already learn and get to know a bit about k6 as well as the performance test itself, here are my key takeaways;
Performance test is one of non-functional testing that need to be executed to get to know and helps us to identify how well our service/application can handle a number of users and transaction that keep growing.
Besides identify the results from our side (to know the response time and etc), we also able to do collaboration with Developer and/or DevOps to monitor the CPU/memory load on the service side to get the full report and know the behavior of how the service was doing if expected load is trying to do some transactions.
Last, there are plenty tools that can be use, such as JMeter, Locust, LoadNinja, and etc. But, k6 is one of them that provide developer-friendly claim so it can be an option to do the performance test. Choose the tools that fit with the test requirement, the team needs, and team knowledge to make it powerful and useful.
