Tomcat Native / OpenSSL in Spring Boot 2.0

Imagine you’re a Spring developer working for company X on a micro-service based platform with services written in Java/Spring Boot and deployed using Kubernetes (Docker). Your various services communicate with one another using JSON over HTTP, making things simple. You assume that all inter-process communication happens over your “trusted” network (think private datacenter or AWS VPC, etc.) and therefore doesn’t need any sort of transport level security.

Now imagine that your company hires a new information security lead who gets to work on his first day auditing your beloved platform — including your method for dealing with transport security, or lack thereof. After doing a quick review, he adds a requirement for you to add TLS for all inter-service transport in your datacenter, citing the fact that data needs to be protected in motion, as well as at rest in order to adhere to standard security best practices.

As you’ve written all of your services using Spring Boot, you realize that all you need to do in order to get SSL up and running is to update your services’ configurations. After having a look at Spring’s documentation, you realize that in order to make SSL work with Spring Boot, you need to deal with Java keystores <ahem…puke>. This isn’t really the end of the world, as you can probably figure something out with containers’ entrypoints that converts PEM-encoded certificates into keystores for you during the deployment of pods, as the rest of the world uses PEM files (NGINX, Apache, etc…) What’s worse is the fact that Tomcat embedded uses JSSE for SSL, which is horribly non-performant.

The Solution(s)

NGINX sidecar

A fairly common way of dealing with SSL termination is to sidecar an NGINX container in your deployed Kubernetes pods which then terminate SSL and reverse proxy requests into your services. As NGINX is written in C and uses OpenSSL natively, it can achieve very good performance when dealing with TLS.

The main downside of taking this approach is that it increases the complexity of your deployments. Whenever new services are created, you must always maintain multiple container pods, which is less than ideal as they can gobble machine resources needlessly (memory/CPU). A potential workaround for which is the creation of “fat” containers (containers which themselves run multiple processes, including an init process which controls sub process lifecycles). Fat containers are generally considered an anti-pattern, however and should only be considered as a last resort. Let’s not forget the increased latency added due to the additional out-of-process hop required as HTTP requests get proxied.

Overall, NGINX has its strengths and is an excellent option for services that are publicly exposed, but isn’t the best option when dealing with internal TLS termination.

Tomcat Native + OpenSSL

Instead of side-carring, what if you could simply tweak your Spring Boot app slightly, and get the best of both worlds — PEM files + Native performance!?

Enter Tomcat Native, Tomcat’s native bindings for APR/OpenSSL. Note: Sample code, as well as setup can be found on GitHub.

Tomcat Native Setup (OSX)

Note that the following setup can be done in a base Docker container… (See GitHub)

1. Install OpenSSL/APR

$ brew install openssl apr

2. Download the latest Tomcat release

3. Extract tomcat somewhere and cd into the bin folder

4. Extract the file tomcat-native.tar.gz

5. cd into tomcat-native-<version>-src/native

6. Configure using the aforementioned OpenSSL

$ ./configure --with-ssl=/usr/local/Cellar/openssl/1.0.2o

7. Copy your built libs to a well known location

$ cp ./.libs/* /usr/lib/tcnative

Spring Boot Setup

Now that you’ve got all of the necessary libraries setup, we’re ready to update Spring’s configuration that will enable Tomcat’s APR connector:

The idea here is that we override the default TomcatServletWebServerFactory in order to override the protocol used by Tomcat’s connector. The Http11AprProtocol will initialize APR, along with OpenSSL properly. In order to start your app properly, you will need to include the following option to Java’s startup:

# This allows Tomcat to find the native libs 
-Djava.library.path=/usr/lib/tcnative

If all goes well, you should see something similar to the following messages emitted after startup:

Loaded APR based Apache Tomcat Native library [1.2.16] using APR version [1.6.3]. 
APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. 
APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] 
OpenSSL successfully initialized [OpenSSL 1.0.2o 27 Mar 2018] 
... 
Starting ProtocolHandler ["https-openssl-apr-8080"] Tomcat started on port(s): 8080 (https) with context path ''

The nitty-gritty (performance)

All tests were run on HotSpot 1.8.0_162 and Tomcat Embedded 8.5.28 with Vegeta to drive load, along with a locally running NGINX on a MacBook Pro. (Note that I also ran the below tests in Java 9 with similar results.)

Cipher suite used: ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!3DES:!DES:!aNULL:!MD5

Note also that I ran a few passes before collecting results in order to “warm-up” Java’s JIT. Another important thing to keep in mind is that I’m intentionally disabling keep-alives in all of my tests. This was done in order to force the TLS handshake process for each test iteration.

Vegeta CMD:

$ echo "GET https://localhost:8080/sample" | vegeta attack -duration=60s -keepalive=false -insecure -rate=100 | tee results.bin | vegeta report

For the impatient

See below for individual test results

Baseline HTTP (no TLS)

Idea here was to skip TLS all together in order to get a baseline from which we can compare other results.

Startup CMD:

$ java -Dserver.ssl.enabled=false -Xms4g -Xmx4g -XX:+UseG1GC -jar target/tcnativeapp-0.0.1-SNAPSHOT.jar

Results:

Requests [total, rate] 6000, 100.02 
Duration [total, attack, wait] 59.99225804s, 59.989998s, 2.26004ms 
Latencies [mean, 50, 95, 99, max] 2.643921ms, 2.832854ms, 3.159403ms, 3.351027ms, 5.25038ms 
Bytes In [total, mean] 12000, 2.00 
Bytes Out [total, mean] 0, 0.00 
Success [ratio] 100.00% 
Status Codes [code:count] 200:6000

JSSE

Just plain ‘ol Java SSL support that you get out of the box when using Spring Boot. This is the setup that you’d get if you followed Spring’s docs directly…

Startup CMD:

$ java -Dserver.ssl.aprEnabled=false -Dserver.ssl.keyStore=`pwd`/target/keystore.jks -Dserver.ssl.keyPassword=password -Xms4g -Xmx4g -XX:+UseG1GC -jar target/tcnativeapp-0.0.1-SNAPSHOT.jar

Results:

Requests [total, rate] 6000, 100.02 
Duration [total, attack, wait] 59.999126055s, 59.989999s, 9.127055ms 
Latencies [mean, 50, 95, 99, max] 8.958652ms, 8.920439ms, 10.156994ms, 11.644574ms, 27.745671ms 
Bytes In [total, mean] 12000, 2.00 
Bytes Out [total, mean] 0, 0.00 
Success [ratio] 100.00% 
Status Codes [code:count] 200:6000

Tomcat Native + OpenSSL

This setup leverages the Native libs built in this guide, along with the various tweaks to Embedded Tomcat’s configuration as outlined here…

Startup CMD:

$ java -Dserver.ssl.certificateFile=`pwd`/target/cert.pem -Dserver.ssl.certificateKeyFile=`pwd`/target/key.pem -Djava.library.path=/usr/local/tcnative -Xms4g -Xmx4g -XX:+UseG1GC -jar target/tcnativeapp-0.0.1-SNAPSHOT.jar

Results:

Requests [total, rate] 6000, 100.02 
Duration [total, attack, wait] 59.99576466s, 59.989998s, 5.76666ms 
Latencies [mean, 50, 95, 99, max] 5.115931ms, 5.31726ms, 5.897839ms, 6.222025ms, 21.364298ms 
Bytes In [total, mean] 12000, 2.00 
Bytes Out [total, mean] 0, 0.00 
Success [ratio] 100.00% 
Status Codes [code:count] 200:6000

NGINX as reverse proxy

In this setup, I setup NGINX as a reverse proxy (guide here). All load was sent through NGINX whose main purpose was to deal with TLS.

Startup CMD:

$ java -Dserver.ssl.enabled=false -Xms4g -Xmx4g -XX:+UseG1GC -jar target/tcnativeapp-0.0.1-SNAPSHOT.jar

Results:

Requests [total, rate] 6000, 100.02 
Duration [total, attack, wait] 59.995316001s, 59.989999s, 5.317001ms 
Latencies [mean, 50, 95, 99, max] 5.364559ms, 5.634727ms, 6.371684ms, 6.789601ms, 26.711814ms 
Bytes In [total, mean] 12000, 2.00 
Bytes Out [total, mean] 0, 0.00 
Success [ratio] 100.00% 
Status Codes [code:count] 200:6000

Conclusion

JSSE is slow!! Overall (95th percentile), it’s about 50% the speed of the APR-enabled variant. Please do take a look at my sample app for guidance on how to set Tomcat Native up for yourself.