Hail to the New King… or Not?
Is there a new leader in a Java town?
Where each journey begins
Let’s imagine for a while that you’re a Java team lead who is going to start a new project. A completely fresh and shiny, beginning from scratch. Cool feeling, right? Pretty interesting business domain, smart guys in the team and a client, who knows what is needed from the beginning. You feel like a hero who will change the world by implementing that using lingua franca in Java universe: Spring Boot. The confidence level is very high since it’s not the first project you’ve been implemented using this Swiss scissor. You know that whatever could you think of, it has been probably already implemented in this stack and just waiting for you to be used for free. It’s a good day you think — great project ahead with usual tooling. Almost the whole team seems to be satisfied with a potential tech stack choice. But there is the one sitting in the corner, who looks like being far from being happy…
New cool kids on the block
His name is Joe. He just took part in one of those fancy Java conferences. And there he saw a presentation of another new and great framework: Quarkus. Probably all of you knows this feeling: a fascination, almost love and definitely the confidence that it beats everything that is currently available on the market. After 2 hours talk he became an evangelist of this new tool and he disagree with your initial choice. There are several points he’s mentioning: ease of developing due to the developer mode, nice set of out-of-the-box features implemented using curated Java technologies and what is the topping on a cake: very efficient resource utilization.
So there is a scratch on the surface of your perfect plan. The team seems to be confused and they are waiting for reaction of the team lead. But you’ve been there already. It’s not the first time when you see a new knight in the castle who is shouting: ‘Hail to the new king’. But you know what should be done next: it’s neither sticking to the well-proven stuff nor blindly following a new wave and betting all-in to for a new contestant. Your call is to…
Benchmark
There is no way to judge the developer experience within a short period of time. You definitely have to work with a tool much longer to see whether you like it or not. Initial positive impressions might be ruined after first issues that requires spending a lot of time to analyze them. You know it can’t be a part of your benchmark since it’s very subjective too.
So you’re thinking about the next thing mentioned by Joe: standardized libraries. You’re looking at Quarkus: ok, there is a lot of good stuff in there. On the other hand there is Spring Boot which de facto is a standard itself. After a while you’re realize the tooling is very similar and there are a lot of common libraries used by both of those frameworks. We have a tie in here, so we’re going next.
Last try: performance and resource utilization. This seems to be a good candidate. First of all during the kick-off you heard that solution performance is very important for the business. As always resources (CPU and memory) are implicitly important (more required resources = more money spent. There is no client in the world who says: max everything for my small project; I don’t care with costs). So you’ve nailed it: let’s check performance using simple load tests.
Application features
So now you’re thinking about how to implement a sample application for benchmark. For sure you would like to see some I/O performance since the new app will heavily rely on a database. Also you would like to see how it behaves with regular, blocking method calls, so you’re adding this to your list too. Of course you have to expose an application functionality using REST calls. Also each application have to be Dockerized since a deployment will took place on a Kubernetes cluster. It seems that for now the list is completed and you’re good to go.
This is a sample implementation of your idea using Spring Boot (all other implementations can be found in this Github repository:
@RestController
@RequiredArgsConstructor
public class ExampleController {
private final ExampleService service;
@GetMapping(path = "/api/hello")
public Message hello() {
return service.blockingHello();
}
@GetMapping(path = "/api/cities")
public Cities cityByCountryCode(@RequestParam("country_code") final String countryCode) {
return service.findCitiesByCountryCode(countryCode);
}
}@Service
@RequiredArgsConstructor
public class ExampleService {
private final CityRepository repository;
@SneakyThrows
public Message blockingHello() {
Thread.sleep(100);
return new Message("Hello!");
}
public Cities findCitiesByCountryCode(final String countryCode) {
var cities = repository.findAllByCountryCode(countryCode).stream()
.map(City::from)
.collect(Collectors.toList());
return new Cities(cities);
}
}
Competitors
There are two main competitors in this clash: Spring Boot and Quarkus. But as always the world is not that simple. There are blocking implementations and reactive ones (for Spring Boot we’ve got Webflux, for Quarkus there is Mutiny) in each of them. Another thing are Docker flavors: besides regular JVM Quarkus team is offering native image basing on GraalVM.
Having that in mind you’ve ended with the following list of competitors:
- Quarkus + Jackson + JVM (alias
quarkus-jvm) - Quarkus + Jackson + native image (alias
quarkus-native) - Quarkus + Mutiny + Jackson + JVM (alias
quarkus-mutiny-jvm) - Quarkus + Mutiny + Jackson + native image (alias
quarkus-mutiny-native) - Spring Boot + Jackson + JVM (alias
spring-boot-jvm) - Spring Boot + Webflux + Jackson + JVM (alias
spring-boot-webflux-jvm)
Each application was already built and has a Docker image available under this repository
Benchmark scenario
Gatling was used as a load testing tool. Each testing round consists of 2 strategies of load generation:
- 5 req/s during 10 minutes
- ramping to 8 req/s during 20 minutes
The following code snippet shows the scenario (really basic one, every user just queries two available endpoints).
class QuarkusVsSpringBootBenchmark extends Simulation {
private val warmupStrategy = List(
constantUsersPerSec(1) during (3 minute)
)
private val constantUsersStrategy = List(
constantUsersPerSec(5) during (10 minutes)
)
private val spikeStrategy = List(
rampUsersPerSec(0) to 8 during (20 minutes)
)
private val loadStrategies = Map(
"WARMUP" -> warmupStrategy,
"CONSTANT" -> constantUsersStrategy,
"SPIKE" -> spikeStrategy
)
// rest of the setup ommitted
...
val scn: ScenarioBuilder = scenario("Benchmark")
.repeat(scenarioRepeatCount) {
exec(http("/hello")
.get("/hello"))
.pause(3)
.exec(http("/cities")
.get("/cities?country_code=NLD"))
.pause(2)
}
setUp(scn.inject(loadStrategies.getOrElse(strategyName, warmupStrategy)).protocols(httpProtocol))
}Also there is a bash script provided for starting Gatling with different parameters. Each script run was treated as a test round (for results warmup run is excluded).
#!/bin/bash
// variables setup ommitted
...
echo "Running warmup..."
export BENCHMARK_STRATEGY_NAME=SINGLE
./bin/gatling.sh -rd "warmup ${image_tag}"
echo "Running test 1/2..."
export BENCHMARK_STRATEGY_NAME=CONSTANT
./bin/gatling.sh -rd "test 1/2 ${image_tag}"
echo "Running test 2/2..."
export BENCHMARK_STRATEGY_NAME=SPIKE
./bin/gatling.sh -rd "test 2/2 ${image_tag}"Show time
The whole benchmarking scenario took place on Kubernetes cluster hosted on GCP (3x nodes with the cheapest N1 instance type). Gatling was running on a separate VM (cheapest N2 instance) hosted in the same zone as the cluster. The database was hosted on the Heroku (free plan) just to spice things up and add some latency. Pods were exposed by a service of a type LoadBalancer. Between each round resource limits were adjusted accordingly.
The tables below consist only the best results (lowest p99) of each round for each application. The full lists of results (each round were executed at least 2 times for each application) can be found in Github repository.
Round #1
Initial setup, rather not optimized for an application with such limited functionality. What’s important: for Spring Boot probes delays have to be extended due to a slower time for a first response.
Constant load — /hello
Constant load — /cities
Ramp up load — /hello
Ramp up load — /cities
Thoughts: all candidates looks pretty similar. A big surprise for me is performance of Spring Boot Webflux in ramp scenarios (both I/O and not I/O related endpoints have problems with responsiveness which can definitely exclude network problems related with database communication).
Round #2
Let’s rise the bar a bit higher and reduce available memory a bit. This resulted in further extension of probes delay for Spring Boot.
Constant load — /hello
Constant load — /cities
Ramp up load — /hello
Ramp up load — /cities
Thoughts: this is the place where old champion has failed. Kicked out Spring Boot requests were caused by pods OutOfMemory error and an infinitive restarts loop. For Quarkus cases it’s pretty surprising that non-blocking implementation (which uses reactive Postgres driver under the hood) seems to be slower comparing to the regular one.
Round #3
Spring Boot is dropped due to the results from the last round. Rest of the competitors will be tested with CPU request reduced by a half.
Constant load — /hello
Constant load — /cities
Ramp up load — /hello
Ramp up load — /cities
Thoughts: very similar results of all Quarkus based applications. Again Mutiny based implementations seems to be slower than the regular ones.
Round #4
The most requiring environment for the competitors. Both JVM-based Quarkus apps were dropped due to the startup issues (they were not able to start within the limited probes delay; for a potential new king there were no mercy and delays were not extended).
Constant load — /hello
Constant load — /cities
Ramp up load — /hello
Ramp up load — /cities
Thoughts: last round, last two competitors. This time the Mutiny based implementation wins and passes the test.
So now what?
Results are pretty obvious: in terms of the raw performance and resources utilization there is only one winner. Due to a completely different approach in the implementation of frameworks internals (most of the expensive stuff is done during a compilation instead of runtime backed by multiple proxies) it can lower a bar a lot when it comes to CPU and memory requirements without a compromise on performance. Also ease of native image creation (when you stick with the modules offered by the Quarkus team; otherwise it can be very tricky with third-party dependencies. Also you have to have it might be very time-consuming to create one) shows a great potential for micro-services implementation. You can gain a lot if your workloads require containers to dynamically scale up and down depending on the traffic.
But is it a silver bullet?
One might think now that Spring Boot should be forgotten and abandoned as e.g Java EE. But this opinion would be completely wrong. Raw performance definitely is not the only property to evaluate when it comes to a tech stack selection. Quarkus is still something new and there are many missing pieces there that are available in Spring out-of-the-box (cache implementation using Redis as a store to quickly name one that I was missing). Also some libraries used by Quarkus (e.g: RESTeasy) might not be known by many Java developers.
Spring Boot is also great when it comes to micro-services development and testing (many battle-proven ready to go modules and integrations like Eureka, Feign etc. and great tooling like Spring Cloud Contract). Quarkus, although developed very actively, can’t catch up with years of Spring ecosystem development within a seconds.
You should not follow hype but precisely evaluate the goals you want to achieve and constraints you’re working with.
When it comes to a technology selection it’s always about architecture drivers relevant for a particular project. You should not follow hype but precisely evaluate the goals you want to achieve and constraints you’re working with. As always there is no silver bullet that will solve all your problems. Use your experience and judge what’s more important for you: if your main concern is a raw performance then you should give Quarkus a shot. If you can benefit from its startup time (especially in native mode) then it’s definitely worth trying. But be aware it might not be easy: there might be many pieces missing that existence were obvious in Spring’s world. On the other hand: when you can make a compromise and sacrifice a bit lower resources utilization and get a rich and battle-proven ecosystem of modules and integrations (which means faster time to market) instead of it then Spring Boot is still something great.
PS.: Joe was not able to convince the team to his newest and greatest idea. They decided to go with the Spring Boot at the beginning and leave the doors for eventual future migration. Joe was so sad that he can’t experiment spending customers money that he decided to leave the company and try CV driven development somewhere else.
PS 2.: Try not to behave as Joe. Please experiment in your free time instead of risking others money :)
