Today’s blog post stems from a recent discovery in a customer project. We are maintaining a service that was suffering from a performance problem. The service prepares infrastructural data from various sources and offers them to the outside world, nothing extraordinary. This data is held in memory to ensure very fast access and query response times.
But somehow, the response times were much much slower than expected. For example, the query for receiving all instances of some
Resource A took about 90 seconds to deliver. Looking at the code, nothing justifies this long response time. There were not even any filters applied for this get-all-resources-A query.
The service is written in Java, uses the Spring Boot framework, exposes some REST API via Spring Web MVC and follows the HATEOAS design principle (and uses the Spring HATEOAS implementation for that matter).
Query responses look like this:
"some-data": "i'm valuable data"
"some-more-data": "even more data",
The respective resources contain some metadata and offer references to other resources via link relations (also to themselves via the self relation). The query I mentioned in the beginning is required by some of our automation scripts that actually needs every instance of resource A and all of its relations.
This response contains 1.143.751 such link relations. Well… so, this link generation is better performant.
Spoiler alert: It wasn’t ¯\_(ツ)_/¯
For convenient link generation, the
WebMvcLinkBuilder class provides several methods. We used
linkTo along with
methodOn to infer links automatically by referencing the respective controller method. Really fancy stuff and convenient to use as can be seen in the following example:
However, as nice as it looks, the implementation is very costly. Profiling the application showed significant CPU hotspots in just generating the links, both in the
linkTo and the
methodOn method, the latter one even worse.
That one generates a proxy instance of the controller class, adds some method interceptor to introspect parameters, and does some more reflection stuff in the background you wouldn’t expect. I have to admit, I still haven’t quite understood how this implementation works. There were plenty of attempts by the Spring HATEOAS guys to improve the implementation, e.g. by caching those proxy classes aggressively, however for our case this was not enough.
We had to come up with a different solution.
Iteration 1: Getting rid of methodOn
By using an alternative
linkTo method on the
WebMvcLinkBuilder class, we could already increase the performance by around 50% overall. No joke.
A bit less convenient, as we have to add the resource id ourselves, but already much faster. However, if your resource path contains query parameters, these have to be added here manually:
param1 query parameter cannot be inferred anymore. This actually worked out-of-the-box with the
So, you would have to write something like:
add(linkTo(ARestController.class).slash(instance.getId() + "?param1=false").withSelfRel());
One might guess that query parameters can be added conveniently via the
LinkBuilder. Unfortunately, this is not possible. We are required to add them manually inside the
Already less convenient, but in the profiler we could observe that
linkTo still costs quite a lot of performance.
Iteration 2: getting rid of
linkTo as well
Now we’re skipping
WebMvcLinkBuilder completely for more performance. Let’s examine the implementation for the next iteration.
As an alternative to
BasicLinkBuilder class is provided by Spring HATEOAS. The downside is that the request mapping for the controller must be given explicitly. The
linkTo method in the previous implementation could resolve this part automatically by examining the
@RequestMapping annotation on the controller. At least the servlet mapping can be omitted by using
By using this implementation, the performance increased again, by a factor of around 2,5. Response times were decreased from the initial 90 seconds down to around 20 seconds.
But still, all data for the request is in-memory. Why does the REST request still takes that long? There must be room for more!
Iteration 3: getting rid of
slash as well
After consulting the profiler again, we could examine that the
slash method actually is slightly more costly than it should be. Most importantly, each
slash call creates a new instance of the
LinkBuilder. Don’t get me wrong. In comparison to
linkTo these calls are really peanuts and took some microseconds on average. But peanuts sum up, when you have over a million link relations and three calls to
slash on each.
Therefore, let’s get down to the wires and assemble these links ourselves by using the good old
String baseUri = BasicLinkBuilder.linkToCurrentMapping().toString();
StringBuilder str = new StringBuilder(baseUri);
Well, you’d better write some auxiliary methods to reduce the amount of code required, because that is the price for a solution that performs quite well in this case. Or, if you’d really like to be fancy, how about an annotation post processor that generates fast and reliable code?
With this implementation, the response times were brought down to about 11 seconds (from 90 seconds), all just through optimisations to the resource link generation.
By carefully examining the CPU hotspots by using some profiler, the response time for the specific query could be reduced by the factor of 8 to 9.
Step by step, some of the convenient features for link generation were replaced by hand-crafted solutions. If you are using Spring HATEOAS and your responses contain a lot of self-generated links, I hopefully could give you some useful hints for your refactoring journey.