Approaches to asynchronous processing in Spring-based web applications
Performance is a buzzword in IT, with most approaches to improving a system’s performance being related to improving external factors and rarely its internal implementation.
Some of the common factors that can affect an API’s performance are:
- network latency or its topology
- slow down-streams (another API, a database, an external provider etc.)
- a bad design/architecture
- inefficient implementations of frameworks
This article focuses on the latter point.
Threading on thin ice
HTTP requests can be served either synchronously or asynchronously, depending on the application’s usage, so it’s a design choice.
Debugging code that is asynchronous and can span multiple threads is notoriously difficult. Same applies to logging, information can be easily lost. Implementation wise, it might be more difficult to correctly implement and/or understand async code.
So what’s the catch, why would you choose to not implement your REST layer in a synchronous way?
One might think money, but no, that’s not the only reason, although it certainly can be one of them. A resilient and elastic architecture can “save” an application when load increases or there’s a spike in requests, but that’s not cheap! Most autoscaling solutions have lots of costs associated with them.
Secondly, regardless of the infrastructure choice, an async implementation might improve response times, so users can really get a better product. That’s because when subject to high load, threads are used more efficiently and time spend in queues to get a response are smaller.
Another reason (and perhaps the most appealing one for software engineers) is that there are a lot of applications nowadays that require a shift in paradigms and a different design, namely reactive or stream-based applications. Synchronous-based implementations simply can’t be applied in such cases.
Spring coming to the rescue?
For a Spring-based web application, there are a few frameworks that can be used for exposing and managing your API layer and we’re going to take a look at some of the most popular.
Starting with Jersey, which is a Java EE framework, that is fully compatible with Spring and is the oldest of the bunch. Jersey, be it sync or async, is quite easy to implement and has excellent integration with Spring.
Another popular option is the Spring MVC framework, which has the advantage of being better integrated with Spring and has some Spring-native concepts that you can use immediately. In terms of performance, it’s a bit better optimised for async processing, but the differences compared to Jersey are not noticeable at smaller loads , so if you can’t move to MVC or have specific technical requirements, then Jersey is just as good.
The differences in sync vs. async processing for the 2 frameworks are illustrated here:
In the sync scenario, each request received from the client will be taken over by a thread (1 request 1 thread model), meaning that there is a clear bottleneck in the system, the size and queue of the thread pool. That thread is blocked for the entire request-response cycle.
When implementing the API to handle requests asynchronously we create a setup similar to our next framework, so delegate the processing of the business logic (processing, remote calls etc.) to another thread pool, which can be a Java TaskExecutor. This implementation has the advantage that the request-response thread pool is not busy processing business logic also, but handles only I/O operations. The result is that performance is a bit better and we can say that certain tasks can be async (non-blocking), the downside being that more threads are “spent”.
The last one that I’ve tried to test and look into is Spring Webflux, the latest and greatest that Spring has to offer. It is a complete paradigm shift, starting from the server used (Netty Reactor is default), thread usage and concepts for processing requests (Mono — similar to CompletableFutures, Flux — used for stream processing). Through the Flux and Producer — Subscriber logic, it is well suited for event and reactive driven use cases.
In my case, as the implementation needed to be similar to the other 2 and was just a simple HTTP call, Mono was used. So in a way, not all the power of Webflux is into play.
This time the framework is designed with high concurrency in mind, we have 2 thread pools, one that handles the request-response logic (the IO Selector pool) and the other does pure processing (Worker threads). They’re tied together by the Event loop, which delegates work to a Worker thread and when the work is done it receives a callback from the Worker thread and sends the result to an IO thread, which sends the response back to the client.
All of this is not immune to a system becoming unavailable, blocking tasks can still block worker threads and exhaust resources, so try to implement everything in a non-blocking manner (there are libraries that support non-blocking HTTP and DB calls and more).
A little bit of practice
Talking about resources and performance, let’s see some actual results! Unfortunately, tests were performed on a local machine, so take the results with a grain of salt. The tool for performance tests was Gatling, the external API was mocked via a Wiremock instance running on Docker that was replying with a 50ms delay, each version of the tested apps ran on the same machine.
The test ran for 10 minutes, with a ramp-up from 15 to 150 users per second.
Maybe not the ideal setup, but it’s enough to paint a picture of the results and to let you tweak with some parameters.
Response times show that while Jersey and MVC were quite similar, Webflux managed better altogether. The 99th percentile is not noticeably better, that goes to show that there isn’t much that can be done when dealing with slow external services.
Resource wise, the results are as expected. Memory consumption for sync implementations is much smaller, as internal processing, threads and object instances in the end is lighter. For number of used threads, Webflux is the clear winner, with its better thread management, while for the async implementations of Jersey and MVC the resources needed to handle the load are quite high, which is also expected given the much higher number of threads being created.
All the projects are shared at the end of the article.
It’s worth mentioning that nowadays Spring MVC can use Webflux’s Webclient for HTTP calls, which will clearly improve performance, but I chose to use the now deprecated AsyncRestTemplate to show the results of the baseline framework implementation.
Takeaways
Some points I’ve learned:
- Don’t change Netty or Tomcat defaults unless you really tested everything and know what you’re doing, most of the time defaults are well balanced and thought of
- Performance testing on your local machine can be quite unreliable, Docker helps a bit, but you are still dependant on shared resources. This applies to your pods/containers/VMs, always try to isolate your service from “noisy neighbours”
- Wiremock needed quite a bit of tweaking, it is not the best approach to test heavy load. After about 100 tps it starts to send delayed responses. Would recommend testing against real systems and avoiding mocks as much as possible
- As usual, there are a lot of solutions available for even the simplest of tasks. There are so many possible ways to implement a controller exposing an endpoint which needs to get data from an external API. I am not stating that the implementations shown here are the best available, but they are tested and perform well under load.
Drawing the line
It is important to design your application for the context it will be used in, so don’t overcomplicate unnecessarily, but be open to explore multiple options. I would say that if your technical constraints allow it, go with newer implementations, try them out, compare results and keep your tech debt as small as possible.
