Java frameworks performances on Cloud Run
In the summer 2018, at Google Next in San Francisco, my colleagues and I had the chance to meet Steren. He talked to us about a new product and allowed us to try it in the alpha program. It was the first version of Cloud Run.
Happy and enjoying the honor I packaged my favorite Spring Boot app into a container, deployed it and… WHAT ? 30 seconds of cold start…
Desperate, I continued to test the product, mainly with new apps (mainly python functions packaged into containers) and I even learned Go to obtain good cold start and processing performances.
However, I’m a Java developer since 15 years and I didn’t accept to leave this language because of a first bad experience in serverless world. Moreover, unsafe-type language, like Python, became more and more inefficient and buggy in my team with a lot of newbie developers. And Go was too “new” to hire experienced developer. Thereby, Java remains one of the best tradeoff.
If I have to choose a Java framework, which one has the best performance ?
How to compare frameworks
I’m a senior Java developer, but I don’t know all the tips and optimization of all the different frameworks. By the way, to be the most neutral, I chose to write an HelloWorld HTTP GET endpoint with 2 mains Framework: Spring Boot and Micronaut.
In fact, I wrote nothing, I only took the out-of-the-box examples foreach of them. I won’t describe here how to build these hello world examples, the framework websites are really well documented! I provide the code at the end of the story
And I also wrote an HTTP endpoint in pure Java, without Framework to measure the framework impact on performance.
Each test is packaged into a container and deployed on Cloud Run fully managed.
Spring Boot
Spring Boot is my favorite framework. I’ve developed with it for many years and I still love its handiness and its simplicity. You have a special thing to do ? No problem, a spring component exists for that ! Annotate this, and enjoy !
One of its strength is the autoloading. Add simply a dependency and, automatically at startup, Spring boot scans all the libraries and loads them! Your class is an injectable bean? No need to declare it, annotate it and it’s loaded automatically! Of course, the more dependencies you have, the more files have to be scanned and the longer is the startup time.
This strength in a non-scalable environment, like mine on premise, is perfect. I start my app once, and it stays up until the next revision. Of course, startup time is quite long (20 to 30 seconds), but once a month, it’s totally acceptable!
However, the library scan strength, with the implied extra startup time, is the main pain point in the serverless, auto-scalable and scale-to-0 environment.
Micronaut
I discovered Micronaut in my first session attendance at Next 19 during the spring in San Francisco. I only tried it and didn’t build real app with Micronaut, but it seems to be a good alternative to SpringBoot.
The annotation syntax is similar to Spring Boot and you can also import Spring Boot annotation wrapper in the framework. There are many of the Spring Boot equivalent functionalities. The difference with SpringBoot and the biggest strength of Micronaut is that the libraries scan is performed at the compilation time. Thus, at startup, the app knows which packages to load without having to scan all the files.
Therefore, the startup time is better than with Spring Boot but also, I guess (but I haven’t tested), the startup time should be quite stable in time, even if you add a lot of new libraries and files.
Micronaut + GraalVM
Micronaut has also the capability to be package with GraalVM. This AOT (ahead-of-time) packaging improve the startup time and the memory footprint.
I found interesting to try these performances in addition to Micronaut
Java servlet
Frameworks are useful, powerful and really cool for developers. But often, they are cumbersome and can consume a lot of memory and CPU (impacting startup time) for some unused functionalities.
For a simple helloWorld HTTP endpoint, I chose to develop in pure Java with Jetty webserver. Here, no annotation, just Java, inheritance and overriding! I have to do all by my own.
Spring Boot Webflux
Again Spring Boot? Yes. As you could see in the observed results, Spring Boot out-of-the-box example has poor performances. I was annoyed about that, because I really like this framework.
I watched on Youtube a Next19 session with Ray Tsang. He explained how to use SpringBoot on GCP and he offered to reach him on Twitter if we have questions, and I did. Ray is super cool and he helped me improve my Spring Boot HelloWorld performances with webflux and pom.xml tricks. That’s why I also would like to share these improvements.
A word on JIB Plugin
Except for GraalVM packaging, I always use a Dockerfile and JIB for packaging the container to compare the performance.
JIB is a Maven and Gradle plugin developed by Googlers to package easily and efficiently the Java code into a container. You don’t have a Dockerfile to write, only some configuration parameters if you want to customize it.
Performance
Test procedure and measure
All the tests are performed on Cloud Run in the us-central1 region. Container are deployed in a secure mode (allow-unauthenticated option disabled) and with the default parameters (memory, no env var, default service account,…)
The startup time has been tested 3 times for each version. The value is these provided in the Cloud Run logs with the comment “This request caused a new container instance to be started and may thus take longer and use more CPU than a typical request.”. The average value of this 3 tests is the cold start value. 3 can seem few, but in reality, Cloud Run performances are quite stable and thus it’s enough for having a sufficient overview of the cold start.
The average response time and the memory usage are taken after a load test performed with Hey. To prevent the scale up of Cloud Run containers, I set the Hey parameters with:
- concurrency to 1 to avoid multiple calls to Cloud Run in the same time
- requests by second to 5 to limit the load and thus the new instance creation.
The load test performs 2500 requests from the Cloud Shell environment. (As you could see, the global latency is about 115ms. Global means from Cloud Shell to Cloud Run, the processing time, and the way back. Except for the cold start, in the Cloud Run logs, the latency is stable and about 6ms. The rest of time is wasted in the network (and I’m in Europe region)).
The average response time is provided by Hey. And the memory usage is provided by Cloud Run console metrics.
Finally, the container size is provided by Google Container Registry.
Observed results
The observed results can be found here
Based on those result, here are some conclusions :
First of all, containers use the same amount of memory in load tests, and the request latency is equivalent.
Second, except for GraalVM, the JIB plugin is more efficient (or equivalent for Servlet) than a Dockerfile built manually. If you haven’t special requirement in the Dockerfile, use JIB first!
I’m not an expert in Dockerfiles, I only used the “official” sample. There is maybe optimisations to improve performance with Dockerfiles.
Third, without any surprise, no-framework usage improves performances, but increases the development complexity and required skills. With frameworks, the performance of Micronaut + GraalVM are incredible for anything Java.
Finally, my fav SpringBoot is quite acceptable when optimized (startup time of 5 seconds) but it’s only a HelloWorld endpoint. I don’t know what the performance will be on a real app, with more dependencies. Not sure that fits all use cases and all requirements.
Container size is only provided for information and doesn’t affect any metrics.
My code is public on Github. You can also use it as a quick start. Dockerfile and cloudbuild.yaml file are present for building the project on Cloud Build. You can also use JIB to build the container and push it, by default to Google Container Registry.
All the build and deployment instructions are in the Readme.md file.
I finally want to thank again Ray for his help and his time in the SpringBoot improvement.
