Performance testing for beginners part 1: First step to the chaos world
Disclaimer:
I must admit: I’m a system admin and a newbie to performance testing, so the content will be very basic. This blogpost will be helpful for ones that start to tread on to the dark realm of performance test (or the ones who intent to do so), to see how I started my journey to the chaos, dark path of performance testing world where everything is hostile and aliens at first. A professional tester may also find it interesting to see the topic under system admin point of view.
Why performance test
Some of you may wonder: which circumstances put a system admin to performance test (or perf test for short)? I want to say FATE or a hostile dark supernatural force, but of course it ‘s not.
My company is using microservices architecture, which greatly reduce the development time. There ‘re new services almost on weekly basis. That made the developers, in order to meet the deadline, must focus on making the service WORK first, and let the PERFORMANCE issues for later (which may never come at all). And of course, the performance defected in their services pile up over time and finally cause problems for business.
As the stability, durability and readiness of services are system admin ‘s responsibilities, we must make sure of services performance before they go to prod. And it surprised me that most of the new services failed the first load test.
At first, when I was assigned tickets for testing performance of some services, I was confused: an issue ask for perf test, another for load test or stress test, but the content seems like the same thing: do this service could serve xxx request/s or xxx concurrent users?
So, I just asked myself: what is the different between perf test, load test and stress test? How do I performing the test? Which tool do I use? How do I know that a service satisfied the business needs? Why some ask for requests per second, and others concurrent users? With that in mind, I started my journey to the dark side, equipped with only desires for knowledge and a little time I hardly gain between mountain of tasks and duties.
What is perf test?
One thing that I see: people usually using term perf test, load test and stress test interchangeably, as if they ‘re the same thing. But actually, they ‘re not. Let ‘s clear this first.
Perf test is a generic name for 6 kinds of test:
There ‘re 6 kinds of test, but I mostly use load test only. A lot of people have been mistaken between load test and stress test, because they appear very similar: both send out a lot of requests to services, but they differ from goals to method:
- Load test: to recognize upper limit (threshold) of a service. Number of request send to service equal to or under threshold.
- Stress test: to study service ‘s behaviour under extreme load - does the service gracefully crash? Does the data be saved to disk before service shutdown? Will it recover after? We could answer those questions by sending a lot of traffic to service, more than it can handle and observe.
Sometime people ask to load test a service to see if it could satisfied xxx concurrent users (ccu). Don’t accept that kind of request, ask them to convert ccu to rps. From a system point of view, only throughput (rps, tps …) could be measured. Because user behaviors ‘re unpredictable: an user could generate a lot of requests while another does nothing at all - there ‘s no such throughput unit like users per second.
Choosing the right tool
One of problems I faced is: which tool do I choose? Of course, It should be free, open source and easy to use with short learning course. And I tried several ones:
- ab: an old command line tool from apache, simple and effective. But it has a limitation: it only uses a single CPU core, which limits the number of request it could send (~ 1k3 rps on my VM). Example of an ab command:
$ # Example of an ab command:
$ ab -n 400 -c 20 -t 40 http://test-api.coccoc.com- artillery: it leaves me a very good first impression: the scenarios could be written with yaml and only took some mins to write a simple one. At moment, I felt like King Arthur when he put his hand on Excalibur: This is it!! The weapon I used to long for, the one that fit me the most. People usually say the first impression is the last, but it ‘s not true with this case. Turned out my Excalibur is only a decayed stick with beautiful coasted, and will be easily broken on the first impact. The tool, running on my VM, could not send out more than 100 rps, which is worse than I could imagine. But a stick still has its usefulness, in case you want to send out only a small load.
# Artillery example of an api test
config:
target: "https://test_service.coccoc.com"
https:
pool: 8
phases:
- duration: 20
arrivalRate: 10
rampTo: 500
name: "Warm up the api"
- duration: 300
arrivalRate: 1000
name: "Load test 1000"
- duration: 30
arrivalRate: 1500
name: "Stress test 1500"scenarios:
- flow:
- get:
url: "/api/extension/get-transactions?extension_id=f27c4005-6a16-43ee-8bf5-0c4855f9c4f5"
- wrk: a great command line tool with simple syntax, limited features and great-great performance. It could easily send out huge number of request, only consume a little of resources (I could easily generate more than 5000 rps on my VM). While its simplicity limits its feature, it ‘s ideal for the case you want to send huge number of simple requests, for example to test your api.
$ # example of wrk command
$ wrk -t8 -c500 -d3m http://test_api.coccoc.com- Gatling: my boss recommended this tool when I was assigned my first perf test issue, but because it ‘s written in java, I didn’t like it at first sight (every system admin I know hate java). After some bad experiences with ab and artillery, I finally come back to this tool. And it ‘s better, No, I must say much better than I expected. Firstly, gatling could use multiple cores. And it has a cool tool named recorder, which records all your actions on a browser, and generate a corresponded simulation. Gatling ‘s simulation is written in scala, but don’t let it scares you. Gatling document ‘s nice, and it ‘s easy to modify a simulation template for a simple load test. The first simulation only took me 1 morning. Additional, gatling generates a beautiful statistics page to visualize your simulation result.
The lesson here: listen to your boss :grinning:.
// example of a gatling simulation
// Just modify baseUrl and scenario inject and you have your own simulation for simple api load testpackage accimport io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._class acc extends Simulation {val httpProtocol = http
.baseUrl("https://dev-api.coccoc.com")
.acceptHeader("*/*")
.doNotTrackHeader("1")
.userAgentHeader(
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36 Gatling"
)
.disableWarmUp
.disableCachingval getScenario = scenario("accounts root - GET")
.exec(
http("GET request")
.get("/")
.check(status.is(200))
)
setUp(
getScenario.inject(
rampUsersPerSec(100) to 2000 during (10 minutes) randomized)
).protocols(httpProtocol)
}
While googling for the suitable tools, I came across an interesting article: comparing best open source load testing tools. The author divides tools into 2 categories: scriptable and non-scriptable. Both have advantages and disadvantages:
- A scriptable tool is flexible, and you could write complex testing scenarios with it. It requires you must know a language and consumes more memory and of course, more time to learn. artillery and gatling belong to this category.
- A non-scriptable tool is simple, easy to learn and use, tend to consume less resource than scriptable tools. ab and wrk fall to this category.
Which tool you choose base on your needs and what you will test.
On the next part, I will write about the common mistakes that you may encounter while doing perf test and how to avoid them.
