How to do Distributed Testing using JMeter

Have you ever needed to simulate a higher number of virtual users (it is usually done in Stress Tests) than your injection machine’s hardware could endure, even after having increased the Java’s Heap Size on jmeter.properties file and having followed JMeter’s best practices?
If so, you probably went to Google and searched about how to perform distributed tests using JMeter. Of course, I also did it when I faced the same problem but I should confess that I found tons of texts teaching how to do distributed testing using a single machine. It did not even make sense for me since it goes against the Distributed Testing principles (we will see it later on).

I really had a hard time trying to find some article that could explain how to put it to work on multiple machines (master and slaves, calm down, we also gonna see it right away) and moreover, how to make the slaves being able to communicate with their master when you have an active firewall. So as I always, I am gonna tell you how I did it successfully and, with some luck, you will be able to perform it as well. (fingers crossed )

Distributed Testing

If you have a slave running a test on the same machine than the master, it is a Non-Distributed testing approach. Distributed testing consists of having a test script being executed in separate computers, despite this, there is still an important part that we have to add on top of it in order to differentiate it from Parallel Testing (which is also about running test scripts in different machines), iteration. All machines involved in the Distributed Testing environment need to be coordinated and synchronized among themselves, and the master node is responsible for managing that while the slaves will receive a test script copy, execute the load against the target (service, application, database, etc.) and report the results in near real time to the master.

Distributed Testing architecture put it in a nutshell:

• Master: the system running JMeter GUI, controls each slave and receives test execution information (collected metrics, threads/virtualUsers activities, errors, etc.) from them.
• Slave: the system running JMeter-server, receives a command from the master and sends a request to the server which is under test.
• Target: the web server (Database, Application, etc.) under test, get requests from slaves and does its job (parsing, processing, replying, etc.).

Once you are now aware of the basic concepts regarding Distributed Testing, let’s get started!

Preconditions

Before we start, there are some points that need to be verified:

• Make sure that you will have the same JMeter version (always use the latest one) on all machines (slaves and master). Although using different versions could work, however, this practice is discouraged once it can cause unexpected issues.
• Confirm that master and slaves are under the same subnet.
• Assure that firewalls on the operative systems are turned off. In some cases, the firewall may still be blocking the communication among master and slaves. You should disable the Window firewall or Linux firewall. (or you can open TCP ports as we are going to see later on)
• Although not being mandatory, it is recommended to have the same java version on all machines (master and slaves).
• Check that all slaves have the plugins that will be used by the test script that will have its execution triggered by the master node. Master sends the .jmx file to its slaves but it does not send the required plugins.
• Test data files (such as .csv files) also need to be manually copied to the slaves once it is also not done by the master.
• Set JMeter/bin folder path as JMETER_HOME environment variable and jdk1.x.x/bin folder path to the PATH environment variable.

Laying the groundwork

Slaves Setup

RMI Keystore Generation

The first decision you need to make is whether you want to have secure communication (by using SSL protocol) among master and slaves or not. Since JMeter 4.0, the default transport mechanism for RMI will use SSL by default. If you don’t want to use it, then it will be needed to open the jmeter.properties file, uncomment the line where server.rmi.ssl.disable property is located and change it from false to true.

If you choose to be cautious, SSL keys and certificates must be generated by yourself(do not panic, it is a very straight forward process that is being explained in the lines below).

To setup the SLL key/cert pair, simply go to your JMeter\bin folder and execute the create-rmi-keystore file (which is a .bat file if your OS is Windows, and a .sh if you are using Unix). Once you run it some questions will be asked and you can type anything as long as it is accepted by the key/cert pair generation script.

Hint: Type ‘rmi’ as the first answer and later on do not type a password, just hit ENTER when the script asks for a password. By doing this you will avoid needing to change RMI configs on user.properties file.

PS C:\Users\vinicius.correia\Desktop\Tools\apache-jmeter-5.1.1\bin> .\create-rmi-keystore.bat
What is your first and last name?
[Unknown]:  rmi
What is the name of your organizational unit?
[Unknown]:  A
What is the name of your organization?
[Unknown]:  B
What is the name of your City or Locality?
[Unknown]:  C
What is the name of your State or Province?
[Unknown]:  D
What is the two-letter country code for this unit?
[Unknown]:  PT
Is CN=rmi, OU=A, O=B, L=C, ST=D, C=PT correct?
[no]:  yes
Enter key password for <rmi>
(RETURN if same as keystore password):
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore rmi_keystore.jks -destkeystore rmi_keystore.jks -deststoretype pkcs12".
"Copy the generated rmi_keystore.jks to jmeter/bin folder or reference it in property 'server.rmi.ssl.keystore.file'"

Once this process is finished, the generated certificate will be in JMeter\bin folder. You will need to copy and paste it to the same folder in all the other machines that will be involved in the Distributed Testing.

If you decided to turn off the firewalls just skip the next section and skip to Master Setup section.

Configuring the slaves’ firewalls

By default, RMI uses dynamic ports for the JMeter server engine. This can cause problems for firewalls once you would need to allow incoming connections from different ports every time you start the jmeter-server script (.bat for windows and .sh for Unix) on a JMeter slave. In order to avoid using random ports, you will need to set a specific value (or just uncomment the line) for the port to be used by editing server.rmi.localport property on jmeter.properties file.

Now you just need to access your firewalls settings and create a rule to allow incoming connections for the TCP port you have decided to use above.

JMeter/RMI requires a connection from the client (Master) to the server (Slaves). By default, it uses port 1099 but you can change it if you want to. To do that, you just need to uncomment line where the server.rmi.port property is located. After that, you will need to create another rule on your firewall to allow incoming connections for that specific port (being that the default server.rmi.port port or not).

If you are using Unix machines, there are several ways to open a TCP port and because of that not going to cover it here but instead, I will leave you a link to a How-To article about it: https://www.wikihow.com/Open-Ports-in-Linux-Server-Firewall.

Master Setup

Configuring the master node is simpler than doing that for the slave nodes. You just need to open the jmeter.properties file, find the remote_hosts property, uncomment it and type the ip addresses (without port number) of all the slaves that you have set up in the last section.

Starting Slaves and Master

We are almost there, but before executing the test script we just need to initialize the slaves and master. In order to start a slave (or server) you need to go to the JMeter/bin folder, find the jmeter-server (.bat on Windows and .sh on Unix) file and execute it. Do it for every slave machine. If the server starts without errors that is what you gonna see:

Starting the master is pretty straight forward. Just go to jmeter/bin and open jmeter (.jar or .bat on Windows and .sh on Unix). Open a .jmx file and go to Run >> Remote Start All (or Remote Start to select individual machines to run your test script).

You will be able to see whether the test execution has begun or not by taking a look at the master’s GUI or in your slave jmeter-server script console.

Master:

Slaves:

Test result file (.jtl or .csv) are going to be generated on the master machine. All you need to do after the test is finished is to grab the test results and generate your report via JMeter (or do your analysis job using another tool, i.e.: Excel).

Limitations

1. RMI cannot communicate across subnets without a proxy; therefore neither can jmeter without a proxy.
2. Since JMeter sends all the test results to the controlling console, it is easy to saturate the network IO. It is a good idea to use the simple data writer to save the results and view the file later with one of the graph listeners.
3. Unless the server is a large multiprocessor system, in most cases 1–2 clients is sufficient to overwhelm the server.

Useful Tips & Troubleshooting

• In some cases, the firewall may still be blocking RMI traffic.
• Symantec AntiVirus and Firewall
  In some cases, Symantec firewall needs to be stopped from windows services.
  1. Open control panel.
  2. Open administrative tools.
  3. Double click services.
  4. Go to down to symantec antivirus, right click, and select stop.
• Windows firewall
  1. Open network connections.
  2. Select the network connection.
  3. Right click and select properties.
  4. Select advanced tab.
  5. Uncheck internet connection firewall.
• Linux
  1. On Suse linux, ipchains is turned on by default. For instructions, please refer to the “remote testing” in the user manual.
  2. On RedHat (or derivatives), iptables (Netfilter) is turned on by default. Execute “service iptables stop” to stop the Linux Netfilter firewall.
• If you are unable to start test execution on remote machines (slaves running jmeter-server) due to Connection time out errors:

1. Check if the slaves’ firewalls are configured to allow incoming connections on the TCP ports that are configured in the following properties of jmeter.properties: server.rmi.localport and server.rmi.port property.
2. Use telnet to verify that your JMeter master node can connect to the slaves through the ports specified above.
3. If you have run out of options just turn off the firewalls from you jmeter machines.

• If you cannot execute the create-rmi-keystore script because keytool is not being recognized as an internal or external command:

1. Add your jdk/bin folder to your PATH environment variable.

Conclusion

Executing Distributed Testing with JMeter is not as difficult as it might look like and it is a crucial way to generate heavy loads during stress tests. If you followed correctly the process described in this article you will very likely be able to run your stress tests robustly. The usage of Grafana+InfluxDB to display the test data in real-time (or near real-time), the containerization (via Docker, obviously) of the whole test execution process and the integration with some CI tool (such as Jenkins) are good examples about how to evolve your JMeter test execution maturity. Hopefully, I will be able to write about it soon.

If you have any doubt, suggestion or critic please leave me a comment.

Reference: https://jmeter.apache.org/usermanual/remote-test.html