6 Tips for Resolving Common Java Performance Problems
Over the past year, one of our projects has experienced remarkable growth tt. I’m not complaining scaling is a good thing, but along the scaling process, as with any software product, we’ve encountered challenges, particularly in terms of performance, as we’ve scaled up.
Scaling this project can be exhilarating, but has also come with a set of performance issues that demanded our attention. Some of these challenges included resource utilization bottlenecks, occasional memory leaks, and inefficient database access, among others.
In an effort to share our experiences and contribute to the collective knowledge of the Java community. I’ve put together an article outlining some valuable tips and insights. I used Digma (CF) to instrument and resolve the performance issues we faced during our scaling process. You can also use tools like JMH, and profilers, to measure performance and find hotspots, respectively, to find where you need to optimize. I believe that by sharing these learnings, we can help others navigate similar challenges more smoothly.
A brief overview of Java performance problems
Don’t get me wrong, Java is a great programming language, but just like any other language, there are inherent challenges that you may encounter as a Java developer. It’s important to note that these challenges don’t diminish the value of Java but underscore the need for you, as a developer, to be aware of potential pitfalls.
These challenges, often manifested as performance issues, require careful consideration and proactive strategies for optimization. While Java’s strengths lie in its platform independence, robust libraries, and extensive ecosystem, addressing performance concerns is key to fully harnessing its capabilities.
While these challenges highlight the need for careful consideration, they also emphasize
some of the benefits that you can get from optimizing your Java applications. It is also important to note that performance optimization is not a one-off process but rather a process that begins during development, continues through testing and deployment, and remains throughout the application’s lifecycle.
Here’s a short list of considerations from a Reddit post by Martijn Verburg, Principal Engineering Group Manager (Java & Golang) at Microsoft. You should keep in mind when trying to optimize your Java applications.
- Methodology.
Choose a performance diagnostic model such as Kirk Pepperdine’s Java Performance Diagnostic Model, Brendan Gregg’s USE, or Monica Beckwith’s Top-Down. Without a structured approach, optimization efforts may lack direction.
- Performance Goals (SLA/SLO).
Clearly define performance goals, such as achieving a minimum of 1000 transactions per second with a latency of less than or equal to 500 milliseconds at P999 on a Standard_DS_v4 VM.
- App Architecture.
Develop a comprehensive understanding of both logical and physical application architecture.
- Timeboxing / Time Budgeting / Observability.
Implement timeboxing or time budgeting strategies. For example, understand the breakdown of round-trip times, such as ~200ms in JavaScript, 400ms in the JVM, and 300ms in the database. Incorporate observability for insights.
- Math and statistical techniques:
Apply mathematical and statistical techniques, including P99 latency curves, baselining, sampling, and smoothing, to gain a nuanced understanding of performance metrics.
- Understanding of technology stack.
Familiarize yourself with how each layer of the technology stack (JVM, CPU, Memory, O/S, Hypervisors, Docker, K8s) interacts. Understand how each layer surfaces key performance metrics to the next.
JVM Understanding.
Deepen your knowledge of JVM internals, including garbage collection (GC), Just-In-Time (JIT) compilation, Java Memory Model (JMM), and related concepts.
Tools and Tactics.
Stay updated on tools and tactics, as this landscape evolves. Explore load testing, observability tooling for timeboxing, resource monitoring, and diagnostic tooling. Utilize tools like JFR/JMC, GCToolkit, and others for effective performance analysis.”
6 Common Java performance problems
1. Memory leaks
One would ask how this is possible when Java has automatic memory management through its garbage collector. Indeed, Java’s garbage collector is a powerful tool designed to handle memory allocation and deallocation automatically, relieving us from the burden of manual memory allocation and deallocation. However, relying solely on automatic memory management does not guarantee immunity from performance challenges.
The garbage collector (GC) in Java automatically identifies and reclaims memory that is no longer in use, which is an essential aspect of the language’s robust memory management system.
However, even with these advanced mechanisms, it’s still possible for even the most skilled programmers to encounter and inadvertently introduce Java memory leaks. A memory leak occurs when objects are unintentionally retained in memory, preventing the garbage collector from reclaiming the associated memory. Over time, this can lead to increased memory consumption and degrade the application’s performance.
Memory leaks can be tricky to detect and resolve mostly because of their overlapping symptoms. In our case, it was the most obvious of the symptoms: the OutOfMemoryError heap error, which was followed by performance degradation over a period of time.
There are many issues that can cause a memory leak in Java. Our first approach was to establish whether this was normal memory exhaustion(due to poor design) or a leak by analyzing the out-of-memory error message.
We started off by checking the most likely culprits, static fields, collections, and large objects declared as static and potentially blocking vital memory throughout the application’s lifetime.
For instance, in the code example below, removing the static keyword when initializing the list drastically reduces memory usage.
public class StaticFieldsMemoryTestExample {
public static List<Double> list = new ArrayList<>();
public void addToList() {
for (int i = 0; i < 10000000; i++) {
list.add(Math.random());
}
Log.info("Debug Point 2");
}
public static void main(String[] args) {
new StaticFieldsDemo().addToList();
}
}Other measures we took include checking for open resources or connections that may block the memory, thus keeping them out of the reach of the garbage collector. Improper implementations of the equals() and hashCode() methods by writing proper overridden methods for the equals() and hashCode() methods, especially in HashMaps and HashSets. Here is an example of properly implementing equals() and hashCode() methods.
public class Person {
public String name;
public Person(String name) {
this.name = name;
}
@Override
public boolean equals(Object o) {
if (o == this) return true;
if (!(o instanceof Person)) {
return false;
}
Person person = (Person) o;
return person.name.equals(name);
}
@Override
public int hashCode() {
int result = 17;
result = 31 * result + name.hashCode();
return result;
}
}Tips to Prevent Memory Leaks
- If your code uses external resources such as file handles, database connections, or network sockets, make sure to release them explicitly when they are no longer needed.
- Employ memory profiling tools, such as VisualVM or YourKit, to analyze and identify potential memory leaks in your application.
- When using singletons use lazy loading instead of eager loading to avoid unnecessary resource allocation until the singleton is actually needed.
- If your code uses external resources such as file handles, database connections, or network sockets, make sure to release them explicitly when they are no longer needed.
2. Thread deadlocks
Java is a multithreaded language. This is one of the features that make Java a suitable programming language, especially for developing enterprise applications that handle multiple tasks concurrently.
As the name multithreaded suggests, multiple threads are involved, and each is the smallest execution unit. Threads are independent and have a separate path of execution such that an exception in one of the threads does not affect the other thread.
But what happens when threads try to access the same resources(locks) all at once? This is when a deadlock occurs. I have had this experience collaborating on a real-time financial data processing system. In this project, we had multiple threads responsible for fetching data from external APIs, performing complex calculations, and updating a shared in-memory database.
As the usage of this tool grew, we started experiencing occasional reports of occasional freezes. The thread dumps revealed that certain threads were stuck in a waiting state, forming a circular dependency on locks.
In this example, we have two threads (thread1 and thread2) that attempt to acquire two locks (lock1 and lock2) in a different order. This introduces circular waiting, increasing the possibility of a deadlock.
public class DeadlockExample {
private static final Object lock1 = new Object();
private static final Object lock2 = new Object();
public static void main(String[] args) {
Thread thread1 = new Thread(() -> {
synchronized (lock1) {
System.out.println("Thread 1: Holding lock 1");
// Introducing a delay to increase the likelihood of deadlock
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
synchronized (lock2) {
System.out.println("Thread 1: Holding lock 1 and lock 2");
}
}
});
Thread thread2 = new Thread(() -> {
synchronized (lock2) {
System.out.println("Thread 2: Holding lock 2");
// Introducing a delay to increase the likelihood of deadlock
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
synchronized (lock1) {
System.out.println("Thread 2: Holding lock 2 and lock 1");
}
}
});
thread1.start();
thread2.start();
}
}To solve this problem, we can refactor the code to ensure that threads always acquire locks in a consistent order. We can achieve this by introducing a global ordering of locks and ensuring that all threads follow the same order.
public class DeadlockSolution {
private static final Object lock1 = new Object();
private static final Object lock2 = new Object();
public static void main(String[] args) {
Thread thread1 = new Thread(() -> {
synchronized (lock1) {
System.out.println("Thread 1: Holding lock 1");
// Introducing a delay to increase the likelihood of deadlock
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
synchronized (lock2) {
System.out.println("Thread 1: Holding lock 2");
}
}
});
Thread thread2 = new Thread(() -> {
synchronized (lock1) {
System.out.println("Thread 2: Holding lock 1");
// Introducing a delay to increase the likelihood of deadlock
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
synchronized (lock2) {
System.out.println("Thread 2: Holding lock 2");
}
}
});
thread1.start();
thread2.start();
}
}Tips to prevent thread deadlocking
- Lock ordering — Ensure that all threads follow the same order when acquiring locks to prevent circular waiting.
- Implement a lock timeout — If a thread cannot acquire a lock within a specified time, release all acquired locks and retry.
- Avoid nested locking — Avoid acquiring locks within critical sections
- Tips to prevent thread deadlocking.
- Lock ordering — Ensure that all threads follow the same order when acquiring locks to prevent circular waiting.
- Implement a lock timeout — If a thread cannot acquire a lock within a specified time, release all acquired locks and retry.
- Avoid nested locking — Avoid acquiring locks within critical sections where other locks are already held. Nested locking increases the risk of deadlocks.
3. Excessive garbage collection
Garbage collection in Java is like the behind-the-scenes hero that manages memory for us. It automatically cleans up objects that are no longer needed, making our lives as developers a lot easier. While this automatic garbage collection provides convenience to developers, it can come at the cost of CPU cycles devoted to garbage collection which can affect application performance.
Apart from the typical out-of-memory error, you may experience occasional application freezes, lags, or application crashes. And if you’re using the cloud optimizing your garbage collection process could save you lots of computing costs. A case example is a company called Uber that was able to save 70K Cores Across 30 Mission-Critical Services by using a highly effective, low-risk, large-scale, semi-automated Go garbage collection tuning mechanism.
Tips to prevent excessive garbage collection
- Log Analysis & Tuning — Identify patterns such as full garbage collection cycles or long pause times.
- Evaluate and switch between different garbage collection algorithms. Consider JDK algorithms like Serial, Parallel, G1, Z GC, etc.
- Choose an algorithm based on your application’s workload and performance characteristics. Switching to a more suitable GC algorithm can reduce CPU consumption.
- Optimize code to reduce excessive object creation. You can use memory profiling tools like HeapHero or YourKit to identify areas generating excessive objects. Implement object pooling to reuse objects and reduce allocation overhead.
- Modify heap size to impact CPU consumption during garbage collection. Consider Increasing heap size to reduce the frequency of collection cycles or decreasing the heap size for applications with a low memory footprint.
- If you are running on the cloud, consider distributing the workload across multiple instances. You can increase the number of container instances or EC2 instances to better utilize resources and reduce strain on individual instances.
4. Bloated Libraries and Dependencies
Build tools like Maven and Gradle have revolutionized the way we manage dependencies in Java projects. These tools provide a streamlined way to include external libraries and simplify the process of project configuration. However, with this convenience comes the risk of bloated libraries and dependencies.
There is even a whole study published back in 2021 revealing that there are dependencies packaged with the application’s compiled code that are actually not necessary to build and run the application. You can find the paper here.
Software projects have a tendency to grow quickly as a result of bug fixes, new features, and new dependencies. Sometimes, projects can grow out of proportion and exceed our ability as developers to effectively maintain them. They can also introduce security vulnerabilities and additional performance overhead.
If you’re in this situation, I’d advocate looking into ways that you can debloat your application of unused dependencies and libraries as one of the remedies.
There are several tools within the Java ecosystem that I have found used for managing dependencies. Some of the most common ones include the Maven dependency plugin and the Gradle dependency analysis plugin, which are pretty good at detecting unused dependencies, used transitive dependencies (which you may want to declare directly), and dependencies declared on the wrong configuration (API vs implementation vs compileOnly, etc.).
You can also leverage other tools like Sonarqube and JArchitect. Some modern IDEAS such as Intellij, also have decent dependency analysis capabilities.
Tips to prevent bloated dependencies in Java
- Dependency auditing — Perform regular audits to identify unused or obsolete libraries. Tools like the Maven Dependency Plugin or Gradle’s dependencyInsight can assist in dependency analysis.
- Version control — Keep dependencies up-to-date. Utilize version control systems to track changes in dependencies and manage updates systematically.
- Dependency scopes — Leverage dependency scopes (compile, runtime, test, etc.) effectively. Minimize the number of dependencies in the compile scope to reduce the size of the final artifact.
5. Inefficient Code
No developer sets out to write inefficient or suboptimal code intentionally. However, despite the best intentions, inefficient code can find its way into production for various reasons. It might result from tight project deadlines, limited understanding of the underlying technologies, or evolving requirements that force developers to prioritize functionality over optimization.
Inefficient example
String result = "";
for (int i = 0; i < 1000; i++) {
result += "example";
}Improved example
StringBuilder sb = new StringBuilder();
for (int i = 0; i < 1000; i++) {
sb.append("example");
}
String result = sb.toString();Inefficient code can lead to increased memory consumption, longer response times, and decreased overall system efficiency. Ultimately, inefficient code can degrade user experience, increase operational costs, and limit application scaling to handle increased loads.
Getting rid of inefficient code starts with identifying patterns that indicate inefficiency. Some of the patterns I’m always on the lookout for include nested loops without proper exit conditions, unnecessary object creation and instantiation, excessive synchronization, inefficient database queries, and many more.
Tips to help you write efficient Java code
- Refactor and modularize your code to avoid unnecessary duplication
- Optimize I/O Operations — Use asynchronous or parallel I/O operations to prevent blocking the main thread.
- Avoid unnecessary object creation, especially in performance-critical sections of your code. Reuse objects where possible and consider using object pooling techniques.
- When building strings, use StringBuilder instead of concatenating strings with the + operator. This avoids unnecessary object creation.
- Use Efficient Algorithms and Data Structures — Choose algorithms and data structures appropriate for the task to ensure optimal performance.
6. Concurrency problems
Concurrency issues emerge when multiple threads access shared resources concurrently, often leading to unexpected behavior.
If you’ve been in the coding game for a while, you’ve likely faced the frustration of emerging issues throughout the development cycle. It’s a real challenge to spot and address these issues effectively.
And let’s be honest, they tend to linger around, haunting your application if you don’t have clear visibility into real-world performance. The struggle is even more real when dealing with complex distributed systems. Without proper insight, it’s tough to make informed design decisions or assess the impact of code changes.
That’s where Continuous Feedback steps up to help. The Digma plugin has been fundamental on this front, allowing us to constantly sniff out various issues throughout the development cycle and providing valuable insights in real time. But let’s not get lost in the background details. Let’s dive into the specifics of what we’ve been up to.
Recently, we noticed some unexpected behavior and hit a roadblock with performance in one of our backend services. So, what did we do? We decided to put Digma to the test, aiming to identify the issue and zero in on the root cause.
In a traditional scenario, such an investigation would have required a backlog item to be created, prioritized, assigned, and hopefully not pushed aside because of some urgent matters. Luckily, using existing observability data, Digma was able to show a very specific analysis of the concurrency issue we were facing.
Digma’s analysis went beyond just identifying the issue but also helping us resolve it by precisely pinpointing the root cause of the scaling problem.
Represented by the orange line in the graph, which runs parallel to the overall execution time, the root cause corresponds to a query call triggered during request handling in a specific endpoint.
Both lines exhibit a similar pattern, indicating that the scaling issue is specifically affecting that particular query, which then propagates to the endpoint. After identifying the root cause, we examined the problematic span and found an inefficient query execution plan.
We addressed the underlying problem by refactoring the query and adding missing indexes, swiftly resolving the scaling issue.
Tips to help prevent concurrency problems in Java
- Use atomic variables — The java.util.concurrent.atomic package provides classes like AtomicInteger and AtomicLong, enabling atomic operations without explicit synchronization.
- Avoid sharing mutable objects — Design classes to be immutable whenever possible, eliminating the need for synchronization and ensuring thread safety.
- Minimize lock contention — To minimize lock contention, use fine-grained locking or techniques like lock striping to reduce competition for the same lock.
- Utilize the synchronized keyword to create synchronized blocks or methods to ensure that only one thread can access the synchronized code block at a time.
- Use Thread-Safe data structures — Java’s java.util.concurrent package offers thread-safe data structures like ConcurrentHashMap, CopyOnWriteArrayList, and BlockingQueue to handle concurrent access without additional synchronization.
Conclusion: resolving Java performance problems
I hope you found this discussion resourceful. Remember not to fall victim to premature optimization. It is also crucial to recognize that not all sections of code contribute equally to an application’s performance. The key lies in identifying and prioritizing the critical areas that have a substantial impact on performance. Digma excels at this by allowing you to make your optimization efforts more guided and less random. In other words, Digma analyzes the runtime data to help you identify those pieces of code that affect the system’s ability to scale the most.
