How to Find API request/response details from a Heap dump in wso2 APIM

In this blog, I’m going to discuss a very simple but important thing for troubleshooting high heap memory, CPU spinning on wso2 APIM 4.1.0[1].

What is Heap memory

This is some sort of a big topic. But if we consider the basic concept,
Heap memory is a part of memory allocated to JVM, which is shared by all executing threads in the application. It is the part of JVM in which all class instances are allocated. It is created on the Start-up process of JVM.

In simple words, this is the place where all the runtime objects are stored.

What is CPU and CPU spinning

CPU is short for Central Processing Unit. It is also known as a processor or microprocessor. It’s one of the most important pieces of hardware in any digital computing system — if not the most important. Inside a CPU there are thousands of microscopic transistors, which are tiny switches that control the flow of electricity through the integrated circuits.

CPU Spinning occurs when a certain thread gets into either an infinite loop or is running a computation-intensive operation that takes a long time. In both cases, the thread is “hogging” the CPU and the operating system cannot preempt and switch to run another thread of the same process (as it usually does).

We can say that High Heap and CPU spinning is the worst nightmare that all most all applications can experience. I think most of you will agree with me.

Performance is a very critical attribute of software like wso2 APIM. Because API management servers should serve API requests without any hesitation and that is their main TASK. At the same time serving APIs will be the vulnerable point of API management servers. Because there can be thousands of transactions in 1 second. From those transactions, there can be high data flow, high rate of runtime object creations and discard, high rate of external-internal network calls, etc. Ultimately this burden will be on the resource level of the server (such as memory, and CPU). That’s why any issue on memory and the CPU is going to be the worst nightmare for the APIM servers.

As I have mentioned, the APIM server’s main task will be managing transactions (AKA API requests/responce). Hence the requests and responses are acting a vital role in the performance of the APIM servers. That’s why it is very essential to find the requests/responses details from a heap dump when troubleshooting most of the CPU spinning as well as High heap scenarios. Then we can either try to reproduce the issue using those details or correct the faulty request/response.

Under this topic, you have to remember one thing. In wso2 APIM, those request and responce details will be stored in org.apache.http.nio.reactor.ssl.SSLIOSession [2] (HTTPS)
org.apache.http.impl.nio.reactor.IOSessionImpl [5] (HTTP)

Let’s Back to the main topic.

You can capture a heap dump using the below command

jmap -dump:format=b, file=<file-path> <pid>

Now we need to dig into the heapdump. In this blog, I’m going to use 2 tools.
- Jprofiler [3]
- Memory Analyzer (MAT) [4]

Jprofiler

First, you need to open the heap dump using jprofiler. This is the first look at the heap dump in the Jprofiler

In the Jprofiler, we can find the URLs, headers, and status codes of the request and responses. You can find step by step below.

I’m considering the HTTPS connections first. As I have mentioned, all details resides under SSLIOSession

Therefore, you can search the SSLIOSession using the search bar at the bottom and hit enter. You can see the object SSLIOSession and right-click on it. Select “Use Selected Objects”

Select the Outgoing reference and hit OK. It will list all the SSLIOSession inside this dump

Now you can see the existing list of SSLIOSessions

Now we need to find the request/responce. You need to navigate the below location by expanding one SSLIOSession.

SSLIOSession → session → attributes → m → table

There are a few elements. You can find the http.request and http.reponce inside 2 of those elements.

But sometimes there can be some error scenarios in which we can’t find the request/responce using the above method. If you encounter such a case then you can use COONECTION_INFORMATION element in the same location (SSLIOSession → session → attributes → m → table). You need to navigate a little bit further into the COONECTION_INFORMATION.

value → sourceConfiguration → sourceConnections → busyConnetions → elementData → element → session → session → attributes → m → table

Again there are few elements and http.request and http.reponce are inside 2 of those elements similarly.

Please note that those requests/responses are current busyConnetions inside the whole server related to this transport and not related to this particular session.

Now you found the http.request/http.reponce using any of the above methods. You can dig into those object and there will be lots of information regarding request and responce.

Let’s say we need to find the number of occurrences of the same status code, same URL, same remote address etc. Because those details are vital in a troubleshooting session. To fulfil that, we can do a quick search as below.

Right-click on the attribute that we need to search and then select as below.

Right click on code=200

Then select the configuration according to the requirement and hit OK. You can find a list of SSLIOSession which are fulfill the logic.

Finally, We could find lots of information from the Jprofiler. But there is one important attribute that is missing. It’s the payload. In the Jprofiler the payload is in the binary format. Hence either we need to decode it or use another tool.

Since decoding is a bit complicated task, I’m using Memory Analyzer (MAT).

Memory Analyzer (MAT)

First look of a Heapdump in the MAT

In MAT, we need to use Object Query Language (OQL) to search the objects as below.

The request and responce details will be resides under,

SSLIOSession → appBufferStatus → inbuf/outbuf

You can copy the request/responce to a file from

inbuf/outbuf → buffer → hb

This is a sample copied responce from the above method

HTTP/1.1 200 OK
activityid: 6d33dcb7-ffd5–4d1e-b2aa-9bd77a931b15
Access-Control-Expose-Headers:
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET
Sozu-Id: 01G7DCJT734WJ906NJHBJ9D3D2
Access-Control-Allow-Headers: authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,apikey,Internal-Key,Authorization
Content-Type: application/json; charset=UTF-8
Date: Thu, 07 Jul 2022 22:30:42 GMT
Transfer-Encoding: chunked
Connection: keep-alive

12
{“hello”:”selaka”}
0

In the same way, you can search HTTP communications using org.apache.http.impl.nio.reactor.IOSessionImpl

In Jprofiler

IOSessionImpl → attributes → m → table → elements →http.request/http.responce

Yes, It is very simple Using MAT. Then Why do we bother to use JPORFILER?

Jprofiler is very CUTE and MAT is very ugly. Isn’t it?.

Just kidding, The actual cause is, in Jprofiler the searches are far easier than MAT. Even if we could find the requests and responses very easily, Search is very essential in troubleshooting.

That’s why I’m using both tools.

That’s it. Hope you gain some knowledge from this. Now there is a whole world inside the HEAP to explore. Let’s dive into the HEAP and explore new things.

1. https://apim.docs.wso2.com/en/latest/
2. https://hc.apache.org/httpcomponents-core-4.4.x/current/httpcore-nio/apidocs/org/apache/http/impl/nio/reactor/SSLIOSession.html
3. https://www.ej-technologies.com/products/jprofiler/overview.html
4. https://www.eclipse.org/mat/
5. https://hc.apache.org/httpcomponents-core-4.4.x/current/httpcore-nio/apidocs/org/apache/http/impl/nio/reactor/IOSessionImpl.html