PERCENTILE, BEST MEASURE FOR RESPONSE TIME
WHAT IS PERCENTILE?
Percentile (common measurement in statistics) splits the given sample group into 100 equal-sized intervals and this provides the data to be analyzed in terms of percentages.
We can tell the usage of percentile with an example: The response time for a HTTP request below which 90% of the response time values lie, is called the 90-percentile response time.
In the following graph, 90% of the requests is processed in 3.0 seconds or less:
WHY DO WE USE PERCENTILE?
If we speak statistically there are a lot of methods to define just how good of an overall experience your product is providing. Averages are used widely. They are really easy to calculate and understand — however they can be misleading.
For example: Let’s admit the average salary in a region in Europe was 1900€. When we are looking closer, we found out the majority, 9 out of 10 people, only earned around 1000 Euros and one would earn 10.000. If you calculate, of course you will see that the average of this is indeed 1900, but we can all agree that this doesn’t represent the “average” salary as we would use the word in day to day live. So now let’s apply this thinking to application performance.
THE AVERAGE RESPONSE TIME
The Average Response Time is the most commonly used metric in performance management. We assume that this shows a “normal” transaction, whereas, this would only be true if the response time is always the same, the response time distribution will be like bell curved.
In a Bell Curve the average and median are the same. In other words observed performance would represent half or more than half (the majority) of the transactions. In real world, most applications have few very heavy outliers; a statistician would say that the curve has a long tail. A long tail does not imply many slow transactions, but few that are magnitudes slower than the standard.
We recognize that the average no longer represents the bulk of the transactions. A better metric by far are percentiles, because they allow us to understand the distribution. A percentile tells us at which part of the curve I am looking at and how many transactions are represented by that metric. To visualize this look at the following chart:
The green line represents the average. As you can see it is very volatile. The other two lines represent the 50th and 90th percentile. As we can see the 50th percentile (median) is rather stable but has a couple of jumps. These jumps represent real performance degradation for the majority of the transactions.
The 90th percentile (this is the start of the “tail”) is a lot more volatile, which means that the outliers slowness depends on data or user behavior. The important one is here, the average is heavily influenced by the 90th percentile, the tail, rather than the bulk of the transactions.
A percentile gives us better sense of our real performance, because it shows us a slice of my response time curve. For exactly that reason percentiles are perfect for automatic baselining. If the 50th percentile moves from 500ms to 600ms I know that 50% of my transactions suffered a 20% performance degradation. You need to improve that, it is clear.
HOW TO CALCULATE PERCENTILE?
We will tell how to percentile calculated with a simple example
If we have a group of complex number that we need to sort them from 1 to 10. After sorting the array becomes as following:
1–2–3–4–5–6–7–8–9–10
%70’s numerical value is 7;
%80’s numerical value is 8;
%90’s numerical value is 9;
As you can see, calculating percentile is very simple. In Loadium, you can view percentiles in Summary Report after running load test.
Percentiles are also great for performance tuning. For example let’s admit that generally something within your application is too slow and you need to make it faster. In this case you need to focus on bringing down the 90th percentile. This would ensure that the overall response time of the application goes down.
In another example let’s say you have too long outliers you need to focus on bringing down response time for transactions beyond the 98th or 99th percentile.
You too could have seen that many applications that have perfectly acceptable performance for the 90th percentile, but with the 98th percentile being magnitudes worse.
We could not make the same observations with averages, minimum and maximum, but with percentiles they are very easy indeed.
CONCLUSION
Averages are inefficient because they are too simplistic and one-dimensional. Percentiles are a really great and easy way of understanding the real performance. They also provide a great basis for automatic baselining, behavioral learning and optimizing your application with a proper focus.
In short, percentiles are great!
