Performance and Load Testing with JMeter
ABOUT JMeter
The Apache JMeter™ applications is an open-source software, a 100% pure Java application designed to load testing functional behavior and measure performance. It was originally designed for testing web applications but has since expanded to other test functions.
You can use JMeter to analyze and measure the performance of web applications or a variety of services.
Features
· Open source license: JMeter is free, allows developers to use the source code for their own applications.
· Platform independent: JMeter is a 100% pure Java desktop application. So it can run on multiple platforms
· Full multithreading framework: JMeter allows concurrent and simultaneous sampling of different functions by a separate thread group
· Visualize Test Result: Test results can be displayed in different formats such as a chart, table, tree, or log file
· Easy installation: You just copy and run the *.bat file to run JMeter. No installation is needed.
· Multiple testing strategies: JMeter supports many testing strategies such as Load, Performance, Stress, and Functional Testing.
· Support multi-protocol: All basic protocols such as HTTP, JDBC, LDAP, SOAP, JMS, and FTP are supported by JMeter
Requirements
JMeter has minimum system requirements.
Java Version
JMeter is compatible with Java 8 or higher. You can’t start Jmeter if you don’t have Java installed in your system.
Operating Systems
JMeter is a 100% Java application and should run correctly without operating system dependency.
Why Is Jmeter Needed?
When we need Load, Stress, or Performance testing, we can use Jmeter. In this way, we can handle more than one need with a single tool.
Knowing the Performance, Load, Stress or Functional Testing will help us understand our needs.
Performance Testing:
Performance Testing allows us to test whether the applications work within the planned performance criteria. Each application should have some performance criteria. They reflect our approach for the quality of the product.
Load Testing:
It is the type of test that gives information about how much load the system can carry, while still working with maximum performance. It is done to understand how much load the system can tolerate. Thanks to the Load Testing, it is easier to determine the load balancing servers or the necessary hardware required by a system whose performance test has been completed.
Stress Testing:
The Stress Testing is a type of test, performed to observe how the system behaves under a load above the planned maximum capacity, and most importantly, at what point it breaks. The expected behavior is that the application remains stable in producing accurate results after the test is over.
Functional Testing:
It is the technique used to test the features and functionality of the system or software. Functional Testing looks for answers to questions such as “can the user do this” or “does this feature work”.
How Does JMeter Work?
JMeter simulates a group of users sending requests to a target server, and return statistics information of the target server through graphical diagrams.
JMeter Components
In this section, instead of discussing all JMeter components in detail, we will include the most important JMeter components that will allow us to get started quickly.
Thread Groups
Thread Groups is a collection of Threads. Each thread represents one user using the applications under test. Basically, each Thread simulates one real user request to the server.
- Set up a number of threads
- Set up a ramp-up period
- Number of times test execute
Elements of Thread Groups:
Ramp-up Time
Assume that the applications we offer in our test are wanted is to be tested for 1000 users. It is not very realistic to let all 1000 users in the system at the same time. Ramp-up Time determines how many seconds 1000 users will be included in the system by the test tool.
When the Ramp-up Time is given as 20 seconds for 1000 users, 50 users will have entered the system at the end of the first second.
Loop Count
Loop Count is the total number of entries of each test user into the system. For example; if we get 10K users into the system in 1 minute and let the finished user enter the system over and over again (10 times in total), we have provided 100k users.
You can access:
Right-click on the “Thread Group” > Add > Threads > Thread Group
Think Time
It takes some time for real users to use it with a mouse or keyboard between two test steps. This time can be simulated with Think Time, which is determined between steps for virtual users in load and performance tests.
You can access:
Right-click on the “Thread Group” > Add Think Times to children
Sampler
JMeter components are needed to interact with the system to be tested.
Below you can see the most used sampler types.
HTTP Request: This sampler lets you send an HTTP/HTTPS request to a web server. Consider the example below. JMeter sends an HTTP request to the Google website and retrieves HTML files or images from this website.
Access Log Sampler: This sampler allows you to read access logs and generate HTTP requests. The log could be image, Html, CSS…
FTP Request: Let’s imagine you want to performance test an FTP server. You can use an FTP request sampler in JMeter to do this task. This controller lets you send an FTP “download file” or “upload file” request to an FTP server.
JDBC Request: This sampler lets you execute Database Performance Testing. It sends a JDBC Request (an SQL query) to a database.
Mail Reader Sampler: The Mail Reader Sampler can read (and optionally delete) mail messages using POP3(S) or IMAP(S) protocols.
SMTP Sampler: If you want to test a mail server, you can use the SMTP sampler. This sampler is used to send email messages using the SMTP protocol.
Listeners
Shows the results of the test execution. They can show results in different formats, such as a tree, table, graph, or log file.
You can see the most used Listener types below. The type of Listener you will use depends on what you are testing and the purpose of the test.
View Result Tree
View Results in Table
Summary Report
Thanks to the Listeners, we can see the connection time to the server “Connect Time”, the loading time of the page “Load Time”, the size of the HTTP response “Size in bytes”.
Configuration Elements
Configuration elements can be used to set up defaults and variables for later use by samplers.
CSV Data Set Config
Suppose you want to test a website for 100 users signing-in with different credentials. You do not need to record the script 100 times! You can parameterize the script to enter different login credentials. This login information (e.g. username, password) could be stored in a text file. JMeter has an element that allows you to read different parameters from that text file. It is “CSV Data Set Config”, which is used to read lines from a file, and split them into variables.
Jmeter_example.csv file content:
Username1;password1
Username2;password2
The appearance of the CSV Data Set Config component is shown below.
1. Name of the CSV file (actually relative path relative to the location of the JMX file)
2. Variable names are given to be assigned to the columns in the input file.
3. Which character is the separator between the columns in the CSV file.
The most important point to be aware of is that the JMX file where the JMeter Test Plan is saved and the CSV file must be in the same folder, or the relative path must be given according to the JMX file when configuring the CSV file.
HTTP Cookie Manager
Let’s understand this with an example. You used your browser (Firefox, IE…Etc) to browse www.google.com. You log in with your user and password. Your username and password will be stored on your computer as cookies. Next time, when you visit www.google.com, you don’t need to do log in again because your browser will use your cookies as user data to log in.
HTTP Cookie Manager also has the same feature as a web browser. If you have an HTTP Request and the response contains a cookie, the Cookie Manager automatically stores that cookie and will use it for all future requests to that particular website.
HTTP Request Default
This element lets you set default values that your HTTP Request controllers use.
For example,
You are sending 100 HTTP requests to the server google.com
You would have to manually enter server name = google.com for all these 100 requests
Instead, you could add a single HTTP request defaults with the “Server Name or IP” field = google.com
No need to type 100 times!
Assertions
Assertion helps verify that your server under test returns the expected results.
We will see this subject in practice in sample test writing. You can see the assertion types below.
Report Formats and Results Interpretation
JMeter supports the dashboard report generation to get charts and statistics from a test plan.
Open your command prompt and make sure that you are inside the bin folder before running your Load Testing Script.
Run the test in non-GUI mode and save the report as CSV or jtl format.
Windows:
The syntax for running the Script in Command prompt:
Jmeter –n –t [path to test JMX file] –l [path to result in file]
-n: This specifies Jmeter is to run in non-GUI mode
-t: name of JMX file that contains the Test Plan
-l: name of JTL file to log sample results.
Example:
Jmeter –n –t C:\jmeter\gen-report.jmx –l C:\jmeter\gen-report.jtl
Use the following command to generate a report from an existing sample CSV/JTL result file as above.
Windows:
JMeter -g [path to result file] -o [path to report output folder]
-g: [path to CSV file] generate report dashboard only
-o: output folder where to generate the report dashboard after the load testing. The folder must not exist or must be empty
Example:
JMeter -g C:\jmeter\gen-report.jtl -o C:\jmeter\report
After running the script you can open the folder which you have mentioned while running the script, you can see the report has been generated.
Open the Index.html file and you can see your load testing consolidate report.
The reports will contain detailed information about the request, response, pass and failure error, complete graph information.
Best Practices for Jmeter Tests
Jmeter has some limitations especially when it is run in a distributed environment. Following these guidelines will assist in creating a real and continuous load:
1. Do not use the Console to create your load. You can use the console for debugging purposes or to run a small load from it to make sure the script is running correctly. The GUI consumes a lot of memory under heavy load, therefore the console server by itself can not sustain a heavy load.
2. The limit for one load generator is 300/400 threads (Number of Users). Your hardware capabilities, as well as the Test Plan design, will both impact the number of threads you can effectively run with JMeter. The number will also depend on how fast your server is (a faster server makes JMeter work harder since it returns a response quicker).
3. Disable the “View Result Tree” listener as it consumes a lot of memory and can result in the console freezing or JMeter running out of memory. Add listeners appropriately. Keep Result Tree disable when run for a large number of users.
4. Disable all JMeter graphs as they consume a lot of memory.
5. Do not forget to add timers as per your requirement.
6. Always use a relative path for the CSV file if used in the script.
7. Make sure how to use the cookie manager and cache manager along with your requirement.
Advantages and Disadvantages of using JMeter
Advantages:
Performance Testing is crucial to determine that the web applications under test will satisfy high load requirements. It can be used to analyze overall server performance under heavy load.
Apache JMeter testing tool offers the following benefit in Performance Testing:
- JMeter can be used to test the performance of both static resources such as JavaScript and HTML, as well as dynamic resources, such as JSP, Servlets, and AJAX.
- JMeter can discover the maximum number of concurrent users that your website can handle
- JMeter provides a variety of graphical analyses of performance reports.
Disadvantages:
Memory Consumption: JMeter can simulate heavy load and visualize the test report. This may consume lots of memory and can lead to out of memory under heavy load.
Web applications only: JMeter is a good tool for testing web applications but it is not a suitable tool for testing desktop applications.
Lack of support for JavaScript: JMeter is not a browser, so it cannot run JavaScript in web applications. It has limited support for processing JavaScript or Ajax, this may affect the accuracy of the simulation.
Thank you for your interest. Hope it was useful. If you want more information and do practice about load testing with JMeter, just wait for the next post which would be quite useful.
