GraalVM Native Image: Spring VS Quarkus

In 2018, Oracle announced the 1.0 release of GraalVM.

GraalVM is a universal virtual machine for running applications written in JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Groovy, Kotlin, Clojure, and LLVM-based languages such as C and C++. [source]

One of the most interesting features for me is compilation to native binary (in GraalVM usually called Native Image) for JVM-based applications. Native Image means the compilation will be standalone, does not require JVM (like C, C++, or Golang), but it has trade-off (JIT vs AOT) and limitations.

Spring Framework, one of the most mature and most popular frameworks in JVM world, currently doesn’t have native image compilation feature (out of the box) yet because of the limitations I mentioned earlier. In mid 2019 spring-graalvm-feature was developed to support native image compilation for Spring Boot (Spring Framework 5.2), and, in September 2019 officially joined at Spring Projects Experimental Project. I’m very interested to try, but it requires huge memory for build process, and unfortunately my previous laptop RAM was only 8GiB RAM. Good news, this commit make memory usage smaller.

Some new Java frameworks are already have native image compilation feature out of the box, such as Quarkus, Micronaut, and Helidon. A few months ago I interested to try Quarkus, actually 1.0 was released last November, but I didn’t have much free time to try. A few months ago, after a few days of trying Quarkus, I was curious to benchmark Quarkus with Spring (sping-graal-native), but after several experiment with spring-graal-native, I think it’s still painful, even for simple HTTP API and JPA (with mainstream SQL Database), my issue so far #1, #54, #55, #93, #197. No wonder, it’s 0.6.0, maybe wait for the next version or official support at Spring 5.3 is good idea.

Couple weeks ago, spring-graal-native 0.7.1 has been released, many issues have been closed and I have free credits on GCP, so, lets try again and do some benchmark. I know it’s unfair comparing spring-graalvm-native right now, probably let’s just say this is an “experiment report”.

Application

This benchmark is done to four type of applications

1. Spring Boot with WebMVC
2. Spring Boot with Webflux
3. Quarkus
4. Quarkus with Spring API Extension.

All of those are just simple CRUD application over HTTP with PostgreSQL database. There are 6 http endpoints

1. Insert
2. Find by id
3. Find by page
4. Find by specific column (filtering)
5. Update
6. Delete.

You can check my repository for source code. Spring Data and JDBC version in Webflux version (number 2) is not reactive. I added Webflux version just for comparison, because Quarkus is Netty based.

For how to compile the applications to Native Image, you can check this documentation for Quarkus and this documentation for spring-graalvm-native. I use hybrid mode for spring-graalvm-native, and custom build script (see compile.sh on every root directory of spring project).

For framework and runtime version, I use Spring Boot 2.3.1 (spring-graalvm-native 0.7.1), Quarkus 1.5.2, GraalVM 2.1.0 Java 11.

Data Model

id *primary key
resource_string *for sample filtering
resource_text *just 3000 random char for payload

Build Process

Build process was done on 4 vCPUs, 16 GiB RAM Virtual Machine, 2.8% RAM usage when idle.

1.Build Time

spring-boot 8 minutes 39 seconds
spring-boot-webflux 8 minutes 12 seconds
quarkus 3 minutes 55 seconds
quarkus-spring-api 3 minutes 57 seconds

After I tried to compile it several times, the inconsistency of the build time can be up to 20–30 seconds.

2. CPU Usage (%) over time

3. RAM Usage (%) over time

4. Peak RAM Usage

spring-boot 10.1 GiB
spring-boot-webflux 9.8 GiB
quarkus 8 GiB
quarkus-spring-api 8 GiB

5. Binary Size

spring-boot 196.6 MB
spring-boot-webflux 182.7 MB
quarkus 69.8 MB
quarkus-spring-api 69.6 MB

Well, result of build time and binary size for Spring is much higher, lets wait (and maybe also contributing) for Spring 5.3 and non-experimental version of spring-graalvm-native.

Startup Time

All of applications started under 1s at 2 vCPUs or 1 vCPU, I think this section doesn’t need to be explained in more detail.

Stress Test

VM Spesification

Database: 6 vCPUs, 16 GiB RAM, SSD (us-west1-b)

Application: 1 vCPUs, 2 GiB RAM (us-west1-b)

JMeter: 4 vCPUs, 16 GiB RAM (us-west2-a)

JMeter VM is at different region because I used free trial account that have 8 vCPUs per region limitation.

Scenario

I use JMeter Random Controller to randomly request to all endpoint for 15 minutes. Before that, I truncate the table and insert 10000 record.

I didn’t do special tuning or something for all of the application, all default (include Spring WebMVC, so, 200 tomcat threads), just setting up number of connection pool. I only did some tuning at VM level, increase max number of open file and enable tcp_tw_reuse.

Throughput Result

Before I run the 15 minutes test, I tried several 60 seconds tests to find most suitable setting (JMeter threads/concurrent users and the number of connection pool) for this kind of VM specification. I use Quarkus for base line, and this is the result.

Based on that result, I think 150 JMeter threads is big enough. For efficiency, let’s just use 2 connection in pool and 1vCPU.

Well, this is the result for 2 connection in pool handling 150 JMeter Thread in 15 minute.

spring-boot 390.1/s
spring-boot-webflux 631.6/s
quarkus 1042.7/s
quarkus-spring-api 871.1/s

And this is average througput/s over time (every 30s).

Resource Utilization (during Stress Test)

Application VM

• CPU Usage (%) over time

Note: As you can see, Application VM CPU Usage for quarkus-spring-api, and all Spring version were decreasing over time. But on DB VM CPU Usage for those versions were increasing (see next section).

• RAM Usage (%) over time

Database VM

• CPU Usage (%) over time

Note: Database VM CPU Usage for quarkus-spring-api, and all Spring version was increasing over time. I have not investigate that further yet. Probably because I used Quarkus version as baseline when finding the number of JMeter Threads and the other can’t handle request with that setting.

• RAM Usage (%) over time

JMeter VM

• CPU Usage (%) over time

• RAM Usage (%) over time

Full source code is available on my Github page.

ard333/native-binaries-benchmark

Are you looking for any information about remote work?
or have a cool resource about remote work?
remotework.FYI is all you need to know about remote work, find and share cool resources right now.