JVM Memory, Mockito and a trip
Most of us are familiar with Mockito. Mockito is an open-source library widely used for unit testing. As its name suggests, it is a mock library for Java that allows the creation of objects of given class or interface. Once created, the mock will remember all interactions. Then, you can selectively verify whatever iterations you are interested in within your unit test.
Load testing is an important phase of the release process of each component at Strands. The load testing phase is at least executed once. This process is a simulation of multiple users/events using a given component at the same time and working with it concurrently. The following phases are identified:
a) Performance testing
We gradually increase the load by adding more and more virtual users/events to the test and checking the performance parameters of the system at any test phase.
The main things we monitor are:
- Latency, average response time
- Throughput, number of processed requests per second
- Error rate
b) Capacity testing
This type of test answers the most common question in load testing: “how many concurrent users can the component handle while maintaining good response time and error rate?” Again, we add virtual users gradually, but in this case, we know the performance criteria in advance and just need to check that they are observed.
c) Stress testing
Every system has a capacity limit. When the load goes beyond it, the component starts to respond very slowly and even to produce errors. When this limit is reached, we check if the component is handling the stress correctly: does it produce smooth overload notifications without crashing? When the load is reduced back to a regular level, the component returns to normal operation retaining the performance characteristics.
d) Endurance testing
This type of testing (also called “soak testing”) is used to check if the system can handle the load for a long time or a large number of transactions. It usually reveals various types of resource allocation problems. For example, a small memory leak isn’t evident on a quick test but it will influence the performance after a long time. For endurance testing, it’s recommended to use changing the periodic load to provoke resource reallocation.
I have given a general overview of the load testing phase to emphasize that it is an important process within Strands. Now that we are on the same page, Let’s go through an imaginary example.
Our manager shows up nervous and starts explaining that one of our clients in the is experimenting an OutOfMemoryError after running the component several hours in a test environment. The application is probably holding too many objects and has run out of memory. The first thing we think is “that’s not possible. We’d run the endurance test for a whole weekend and the JVM heap was all the time within a range.” However, the customer is always right, so the issue has to be some corner case you haven’t covered.
In the following days we focused on the finding out the source of this issue. We tried to reproduce the same environment as the customer:
- Limited the maximum memory to 2GB: -Xmx2GB
- Generate with JMeter the same amount of events/second
- Use similar hardware (CPU, Memory,..)
- Run the jar/war within the same Application Server/Container
Our main goal was to know what kind of objects were consuming large amounts of memory. There are several ways to do that. One approach is to generate Heap Dumps of the application. A heap dump is a snapshot of the heap that can be used to analyze the distribution of objects. The heap dump can be triggered from the command line using jcmp or jmap. If you are profiling the application, the profilers usually have the option to capture the snapshot. Finally, if you are running a Spring Boot application, the Spring Actuators library provides a REST endpoint to generate the heap dump. In order to analyze the heap dump, you can use some tools like Eclipse Memory Analyzer Tool, JProfiler or some online services.
We were using JProfiler as a tool for profiling and analyzing the different heap dumps. After analyzing all heap dumps, we weren’t able to identify any type of Object consuming large amounts of data. The JVM heap was stable all the time within a range. We weren’t able to reproduce the OutOfMemoryError in our environment. The problem must be something else.
After 3 days, the manager came back and informed us that a member of the team had to travel to the client’s offices and try to identify the issue from within the customers’ infrastructure.
We accepted that there was no other choice than traveling there asap, so there we went. Once there, with help from their IT team, we were able to run our tests and profiling tools on their environment and application, finding out some objects consuming up to 1Gb. How could that be? In our monitoring tasks, we’d never detected it. How did that happen? Well, as you can imagine, it turned out we were storing a lot of Mockito objects.
Let’s try to understand how the Mockito object appeared in the source code of the application. The first thing would be to explain how business units are organized in Strands. There are two main teams: Engineering and Professional Services. The Engineering team is focused on product development. This team rarely has contact with customers unless they’re working on the development of a new product. On the other hand, the Professional Services team is in contact with the customer and is responsible for the integration of Strands’ products in its infrastructure. From time to time, this team needs to make some customizations over the default functionality.
How is the customization implemented? Our development is based on the S.O.L.I.D. principles. Our classes are designed with the Dependency Inversion Principle in mind. This design system says that our classes should depend on abstractions, not on concrete classes. Instead of being dependent on the implementation details, our classes are dependent on the interfaces. Thanks to that, if having a class with different behavior is needed, we just have to implement the interface. Let’s say we have the following interface:
The project already provides some implementations: JMSMessageService, KafkaMessageService, MailMessageService, etc. However, this customer asked for a different behavior on that point in the data flow. They didn’t want to send any message. A mock implementation of the contract was needed. An example would’ve been:
What happened, in reality, was the following:
Ok. This is a mock class that doesn’t do anything. However, it was forgotten that the Mockito class keeps track of all the “sendMessage” method calls. As a result, this object keeps growing until an OutOfMemory occurs. Fixing it was a matter of modifying a line of code.
We learnt a couple things:
Always do the full pipeline of testing in the version delivered to the customer, independently of if it had customizations or not.
Add more constrains in Sonar and some custom rules to display violations on the project’s quality dashboard using the Mockito package.
I had the pleasure to the visit the customer and learn a thing or two. In the end, everything worked out fine. When I got there, we identified and fixed the issue in a couple hours, so the meetings were mainly focused on future functionalities, which was productive and exciting. Plus, I had some spare time to watch an NFL game, eat some local burgers and go sightseeing!
