Jan 10, 2019
Load Testing in a Nutshell
A few years ago, I was working on a web application that I was heavily invested in and was super excited for it to go live. After weeks of pushing new features, we were finally ready to launch this bad boy. We wrote a bunch of unit and integration tests and conducted extensive functional testing to make sure everything was working as expected. Go-live day came, and you can imagine we were pretty confident that it was going to be great as we made sure everything was by-the-book for an ideal software development life cycle. But unfortunately, that excitement/sentiment didn’t last long.
As more users accessed the application the slower it became. It was an awful experience for users as they started getting weird errors like the yellow screen of death, timeouts, slow page response, and server errors. CPU utilization was rising steadily and at one point reached 70% on the production database server. We were perplexed by why a thoroughly tested application would perform poorly in the production environment. While doing the postmortem of the incident, a lot of questions relating to performance measurement surfaced.
The million-dollar question was: “How do we know that our application will perform as expected in front of an escalating user load?” The answer is Load Testing.
“Load Testing is the process of putting demand on a system and measuring its response “— Wikipedia
Why is Load Testing important?
Well, the answer is quite simple- to find performance issues before we go live. Load Testing helps us:
- Determine our application’s capabilities by measuring its response time, throughput, CPU utilization, latency, etc. during average and heavy user load. This will eventually help in determining the infrastructure needs as the system scales upward.
- It gives us an opportunity to find strange behavior or surprises when we subject an application to an insane amount of load (stress testing). Strange behaviors include request timeouts, IO Exceptions, memory leaks, or any security issues.
- It also helps us understand if any new code path or database connection introduced any performance bottlenecks.
You may remember what happened to Macy’s website on Black Friday due to heavy user load. Or when The Chicago Board Options Exchange (CBOE)’s website became unavailable just as it launched its first bitcoin futures contracts due to heavy load. There are tons of other examples and tweets of many unsatisfied customers of such events. Now you know the importance of load testing and how important it is for businesses to invest early and often in the practice of load testing.
How to Load Test?
As a Software Engineer, load testing sounded somewhat daunting to me at first, and I assumed it is usually performed by a Performance or QA Engineer. After doing some research I came across a few powerful and easy to use tools for load testing that could easily be integrated into the development pipeline. Some of these tools are Visual Studio Web Performance and Load Test, JMeter, BlazeMeter, Locust, Gatling, Taurus, and The Grinder.
To get a good sense of how we might use them, let’s look at some of these tools at a high level.
Visual Studio Web Performance & Load Test
Visual Studio provides this hidden gem (only in the Enterprise version) to create a Web Performance and Load Test project. It is used to simulate many users accessing a server at the same time so we can identify performance bottlenecks. Following are the steps to create a load test project in Visual Studio:
For the demo I have created a sample .NET MVC Application.
Step I. Create a new Web Performance and Load Test Project.
Step II. After creating the project add a new Web Performance Test.
A Web Performance Test (WPT) is a collection of HTTP requests and responses which are gathered by simulating a user accessing a web application. WPT opens up a web test recorder in the browser and captures HTTP traffic.
We can have many WPTs in our project based on what functionality we want to test. For example, a user navigating to the/Home and /Person controller can be two different WPTs. These WPTs are nothing but building blocks of our load test.
Step III. Once we create a WPT, it’s time to add a Load Test.
After adding a new Load Test project, we get a wizard to select the type of load test: Cloud-based or On-premises. In this example, I will be using the On-premises Load Test. Cloud-based Load Test is very similar to On-premises, except you would need a Visual Studio Team Services (VSTS) account and you’ll have to specify the Azure datacenter from where the load will be generated.
Step IV. Configure Load Test Duration.
In this step we specify the load test duration, which I am setting to 59 seconds. Ideally there should be some warm-up duration, which is used to avoid factoring in the initial startup performance. The final reports we get will not include any performance insights during warm-up time.
Step V. Configure Load Test user load.
After specifying the load test duration, we select a load pattern; e.g. if we want to keep a constant load, or want to gradually increase the load over time. I am keeping it as a constant load of 25 users.
Step VI. Define Load Test mix.
This is an important step in generating your load test. In this step we add different types of tests to our load test mix. I added one WPT that was generated in Step II. We can have several WPTs, unit tests, or coded UI tests in our load test mix.
Step VII. Configure counter sets.
In this step, we add servers that we want to monitor during our load test. In this example I want to monitor my database server. I can also add a Web or API server to my counter set.
This is the last step in setting up the load test before it’s ready to run.
Step VIII. Run and View Load Test summary.
Once we run our load test, it generates a detailed load test summary with average response time, threshold violations, processor time, error information, etc.
Based on the load test summary of demo .NET MVC application with 25 virtual users, we got an Avg. Page Response Time of 1.08 Sec to retrieve /Contact page and 1.09 Sec to retrieve /About page. There were zero failing tests and no threshold violations in my load test. If there were any errors while load testing, then I would fix them and re-run the load tests.
JMeter
Apache JMeter is an open-source, platform-independent, Java-based application designed to load test and measure performance. It is one of the oldest and best (IMHO) load testing tools. JMeter simulates HTTP requests to a target server and shows the performance of the target server via tables, graphs, aggregate report summaries, etc.
We can load test the following protocols with JMeter:
- Web- HTTP, HTTPS (NodeJs, Java, Web API, ASP.NET, etc.)
- Web Services- SOAP / REST
- FTP
- Database via JDBC drivers
For the demo I have created a sample RESTful Java API using DropWizard. Following are the steps to create a JMeter test plan for load testing an application:
Step I. After installing Jmeter, open /apache-jmeter/bin/jmeter.sh which will open the JMeter GUI.
Step II. Add a Thread Group.
Thread Group is a placeholder for different elements, like Sampler, Timer, Listener, etc. Thread group has different configuration settings:
- Number of Threads: where we can specify the number of threads (users) we want to simulate.
- Ramp-Up Period: the amount of time taken by JMeter to get all the threads (users) up and running.
For example:
If Number of Threads =10 and the Ramp-Up period =100 seconds, then JMeter will take 100 seconds to get all 10 Threads up and running.
The first thread will be sent on the 0th second (beginning of the execution) and then each thread will start after 10 seconds (100/10).
- Loop Count: the number of times we want to execute the performance test. We can also execute a forever loop by checking
Forever.
I have configured my test plan for the following settings:
Thread Count = 10
Ramp Up Period (Seconds) = 100
Loop Count = 1
For every 10 seconds (100/10), one Thread / Request will hit the server.
Step III. Add a new HTTP request Sampler.
HTTP Request Sampler lets us send an HTTP request to a web server. In my test plan, I will be making two (GET and POST) HTTP requests to the /hello-world end-point of the demo DropWizard application.
For HTTP Request, we specify web server properties such as protocol, server name or IP, port number, path, method, etc. For myGET request I will select GET method name from web server properties.
For myPOST request, I will select POST method name from web server properties with message body data as {"id": 3,"content": "Hi"}
JMeter also lets us add or override the HTTP Request header by using HTTP Header Manager. We can add Content-Type, Accept-Language, User-Agent, and Accept-Encoding to HTTP Header Manager.
Once the test plan is ready, it’s time to run the test and see our application’s performance.
Step IV. Result summary.
In order to view load test results, we add a Listener. A Listener is a component that shows the results of the sampler that we add to our test plan. In our test plan we have only two HTTP Request Samplers. The results can be shown in a table, trees, graphs, summary report, etc.
Based on the load test summary report of my application we have the following statistics:
Samples: 10 // number of requests sent
Average: 3 sec // average response time of both HTTP Request
Min: 2 sec // minimum response time
Max: 5 sec // maximum response time
Error%: 0.00% // percentage of failed tests
Throughput: 13.3/min // number of requests per unit of time that are sent to your server during the test.
BlazeMeter
BlazeMeter provides an easy-to-use cloud-based performance testing platform. BlazeMeter extends JMeter; i.e. we can run our JMeter (.jmx) files on BlazeMeter. Let’s say we have a JMeter test plan and we want to simulate it with 8000–10,000 virtual users; It will be difficult to run it from a single machine using JMeter — either we run it in a distributed fashion (which is quite a headache), or we use BlazeMeter service.
Following are the steps to execute a test using BlazeMeter service:
Step I. Create test.
Once you have signed up for BlazeMeter, you can create a test by choosing an appropriate test type.
There are different test types to choose from:
- JMeter Test: upload an existing JMeter
.jmxfile. - URL/API Test: when we want to test an API endpoint.
- MultiTest: multiple scenario tests.
Step II. URL/API Test.
For the demo I will be making an HTTP (GET)Request to GitHub API https://api.github.com/{userID}
After adding URL or API endpoint to load test we can provide load test configurations such as:
- Originate Load from: a geographical location we wish to run the load from.
- Users: the number of users to simulate.
- Ramp up (sec): time it should take to get all users/threads started.
- Iterations: the number of times to perform the test case.
- Duration (min): BlazeMeter will use this time to calculate the end time.
- Delay (ms): delay between HTTP requests.
Step III. BlazeMeter Report.
BlazeMeter provides very intuitive and detailed reports. The best part is we can share these reports with anyone with just a URL.
The BlazeMeter Report consists of the following parts:
Summary Report is the main dashboard page of our report. It shows Max Virtual Users, Avg. Throughput, Errors, Avg. Response Time, Location.
Timeline Report shows a graph which lets us overlay KPIs (Key Performance Indicators). We can also drill down deeper and view each use case’s performance metrics.
Request Stats Report is similar to JMeter’s aggregate report. It shows statistics of each request in a tabular format.
Engine Health Report is the test harness itself. It displays the performance indicators received during the test while monitoring JMeter console(s) and engine(s). The major KPIs monitored are CPU, Memory, Network I/O, and Connections.
Error Report displays all errors received during the test run, categorized by labels (pages) and error types.
Logs Report shows what exactly is happening on the engine. We can view and monitor the log of each server used during the test run. Logs availability and server monitoring provide full test transparency.
Wrapping Up
Users today have high expectations when it comes to websites, and performance plays an important role in meeting those expectations. People are less interested in buying products or reading any content from a website that takes a few seconds or more to load. These slower websites are detrimental to businesses and eat up your revenue. Software that does exactly what the customer needs but takes too long to load might never get used.
As a Software Engineer, I have always focused on delivering robust code and making sure everything is covered by unit, end-to-end (to cover happy paths), and integration tests. These tests have become a standard practice for good software delivery, but load tests have been overlooked in many workflows. That is due to a general assumption that they tend to involve complex scripts and require special tools/expertise or hardware resources to run large-scale load tests.
Fortunately, there are many load testing tools out there which are easy to learn and adopt. BlazeMeter has a really good post on which load test tool to use.
In summary, forming an action plan for load testing is quite simple:
- Record user activity for different use cases. I usually gather user activity data from web analytics services such as Google Analytics, App Dynamics, Splunk, or Amplitude. This data is used to model and simulate real-world user behavior.
- Throw traffic at the application for different scenarios.
- Reproduce performance related issues (if any).
- Identify bottlenecks in the application.
- Fix the performance problems.
- Repeat the process until desired performance results are achieved.
Ideally, load testing should be performed throughout the software development lifecycle so that you can catch performance issues early. One of the benefits of doing load testing early in your development lifecycle is that you can establish a performance baseline. For instance, you can use this to determine how many simultaneous users an application can handle. Then, every time you create a new build you can run your load tests and compare with your baseline results. If the application can’t handle as many simultaneous users as the baseline numbers, then you know some bad code was introduced between now and the last load test. You might also find out that your application’s performance is getting better and it can easily support more simultaneous users.
“If you don’t like testing your product, most likely your customers won’t like to test it either.” (Anonymous)
