Hashbash — a comparison of CPU and IO-bound applications in Go and Java across multiple metrics

I love go. Having written this language both personally and professionally for over three years now, I am increasingly convinced that it’s the best choice for writing small, maintainable and performant applications. Go’s intuitive multi-threading patterns, deliberately limited feature-set, and tiny memory footprint make it an excellent choice for a variety of use cases, especially when optimizing engineering and cloud-computing costs is a consideration.

Or at least, that’s what I’ve been trying to convince my coworkers.

I have worked for RetailMeNot (RMN) — a medium-sized tech company based in Austin, Texas — for about five years. A polyglot company, we have applications and tools built in php, python, node, java, and more. Some of my early experience at RMN was as part of our CRM team, rewriting a major codebase in java 8 and spring boot, and that experience has had a profound effect on me.

My mentor at the time, a brilliant and pragmatic man named David, was forging new ground. Python was the language du jour at RMN back then, and the java codebases we did have were largely old and unwieldy dropwizard applications that nobody really wanted to touch. David convinced me, and eventually much of the engineering organization, that java/spring boot was the best choice for the kinds of applications that RMN was writing and maintaining at the time. The success of our re-platform sparked a flurry of rewrites and new services using the technology, and today we maintain on the order of 25 spring boot services.

Java and large frameworks like spring boot have their place. Today, I wouldn’t dream of starting any substantially large project in a language like python or node. In my experience, the end result is never as maintainable as a version could have been using java and spring. That being said, as RMN moves to a kubernetes-based deployment platform and cost-savings on hardware becomes a larger consideration, it has become increasingly obvious to me that using these tools comes with a cost, namely, memory usage.

The shared-compute model of kubernetes has allowed us to reap tremendous cost-savings by running substantially fewer ec2 instances per application than we used to. This is great, but we could be doing even better. The problem is that even our simplest java services use hundreds of megabytes per pod. The result is that more often than not when additional capacity is needed in a cluster, it’s because some java application needs another couple of gigabytes of memory so that its pods can be scheduled.

Here is my thesis then, so listen: while large, complex applications will always demand large, complex frameworks in order to be maintainable, many smaller applications can easily be written without them, and reap tremendous cost-savings in return. It is my opinion that go, a language with static-typing and wonderful high-level libraries, offers a happy middle-ground for writing small applications that use only a fraction the resources of an equivalent java application.

So far my rants have not managed to convince my coworkers, so I’ve decided to conduct an experiment to gather data to back up my assertions. I’ve written two identical versions of an application, one in go and one in java. I will run identical workloads against each application and collect a variety of metrics for comparison. It is my hope that the data I collect will provide the basis for a realistic comparison between these two languages as they perform under both CPU and IO-bound workloads. Let’s get started.

The Experiment

The application for comparison here is hashbash, a web UI/API-server/rainbow table generator. I won’t go too deeply into how rainbow tables work, except to say that they implement an efficient mechanism for reversing digests produced by cryptographic hash functions back to the plaintext that produced them. The name hashbash, incidentally, refers to how this application bashes your hashes, and is definitely not a reference to the popular tree festival that takes place annually on the first Saturday of April in Ann Arbor, Michigan, where I attended college.

Crucially for this experiment, hashbash is split into two components. The UI/API-server does not perform any of the CPU-intensive rainbow table generation or search operations, but simply publishes requests to rabbitmq for the engine to perform these tasks asynchronously. As such, we can separately compare the performance of the IO-bound web applications and that of the CPU-bound engine applications.

I have designed separate experiments for each of these components. All tests will be run on Amazon EKS with metrics collected by prometheus. The configuration for deploying the experiment can be viewed here.

Hashbash Engine

To test hashbash-engine, our CPU-bound application, I generated 5 rainbow tables of various dimensions on each version against a clean database (I use mysql for de-duplicating and storing the generated rainbow chains, which is a little silly, but is also easy). The resulting generation times are summarized below:

+------------------+--------------+---------+---------+
| Chains Generated | Chain Length | Go      |  Java   |
+------------------+--------------+---------+---------+
|        5,000,000 |         5000 | 0:47:28 | 0:55:50 |
|        5,000,000 |        10000 | 1:31:47 | 1:41:47 |
|       10,000,000 |         5000 | 1:32:45 | 1:50:57 |
|       10,000,000 |        10000 | 3:04:27 | 3:23:47 |
|       20,000,000 |         5000 | 2:59:46 | 3:41:44 |
+------------------+--------------+---------+---------+

So, you know, point one for go. The go version of the engine generates rainbow tables a fair bit faster than the java version. I expected this of course; it stands to reason that the compiled go version running on bare metal would be faster than the java one running on the JVM. Right?

I was pleasantly unsurprised until I took a closer look at the breakdown of where each application was spending time in the generation job:

Time spent generating rainbow chains vs writing them to the database

You’ll notice here that the go version does indeed generate the complete rainbow table in 40 minutes less time than the java one. The difference, however, is entirely due to the fact that the java version spends significantly more time writing rainbow chains to the database. My theory on this is that the java version uses the spring batch job framework, which starts a transaction for every chunk of items being processed, serializing and substantially slowing down database writes. The go version does none of this, which is fine for an application like this one, where a few failed writes isn’t going to make much of a difference. This is not an article about the spring transaction model or optimizing mysql transaction isolation levels, so reducing the write overhead in the java version of hashbash is left as an exercise to the reader.

Most surprising here is that the java version actually spends a little less time generating its rainbow chains than the go version. Put another way, Java performs this extremely CPU-intensive task faster than go does. I was kind of giddy when I saw this. Scientific experimentation proved my preconceived notions wrong. The JVM is a beast.

This is all a bit of a digression from my original point, so let’s see how the resource utilization of each application compares. The graph below shows the generation of all five rainbow tables by each application super-imposed on top of one another:

Generating five rainbow tables in go and java

There are several important things to note here:

• The java version is so bogged down waiting for IO that on average it can’t even utilize all 8 cores on the server
• Still, the java version consistently generates chains at a slightly faster rate than the go version
• The java version uses more than an order of magnitude more memory than the go version

Finally to my point. You’ll see above that during the time the go version was generating rainbow tables, up until around 17:00, the memory stayed below 30 MB. In contrast, the java version reaches nearly 500 MB. I know memory is cheap nowadays, but this is the kind of difference that I don’t think can be ignored.

On a m5.xlarge ec2 instance with 16 GB of RAM, for example, if I wasn’t saturating CPU as this application does, I could only safely run around 25 pods with the memory usage of the java hashbash-engine. I could run around 500 of the go version. ec2 isn’t exactly cheap, so while many people are content to eat this cost to use java/spring boot and the admittedly superior out-of-the-box functionality it provides, I have to wonder at what point it stops being worth it.

Now, an informed observer will point out that there are a number of knobs one can turn to optimize the memory utilization of java applications, and that this test probably isn’t totally fair. Fine. While I’ll point out that no garbage collection or memory settings needed tweaking on the go version to confine it to using 30 MB, I will indulge you for completeness.

I performed a number of subsequent tests on the java version, setting various jvm memory settings, getting OOM killed left and right, and adding memory limits to the pods (to which java versions 10+ natively understand and optimize for). The best I could do was to decrease the memory utilization of the java hashbash-engine to around 330 MB:

Comparison of the unlimited vs the limited java versions generating a rainbow table

This wasn’t free either. You’ll notice that limiting the memory usage results in a fairly substantial performance hit, with the limited java version taking about 25% longer to generate the same number of rainbow chains. The difference is, in large part, certainly due to the tremendous increase in garbage collection pause time necessary to keep memory usage down:

Comparison of total garbage collection pause time for the unlimited vs limited java hashbash engine

In summary, even with a fair amount of effort put towards optimizing the memory usage of the java hashbash engine, the go version (which received no such optimization), still uses an order of magnitude less memory. While still more drastic steps can be taken to further reduce the memory usage of spring boot applications, few companies are going to want to spend the time to aggressively optimize a few hundred megabytes away.

Far easier, if you want to reduce memory usage and hardware costs, would be to just use go.

Hashbash Web

To test the hashbash web application, I wrote a simple locust script to generate load. I ran a workload requesting both API responses and static assets for 30 minutes at around 400 requests / second. The results of the test are summarized in the graphs below:

Comparison of http and memory usage metrics for the two hashbash-web versions

Latency and response rate metrics here were collected by the nginx ingress controller, and so latency numbers reflect only the round trip time from the nginx-ingress pod to the application pod and back, rather than from the client. This is fine, as all I really care to measure is the relative time each version of the application takes to service requests.

You’ll notice a couple things here:

• The average latency of the go version is just a little bit better than the java
• The go version 99th percentile latency is a lot smoother because requests more frequently overlap garbage collection pauses in the java version
• Once again, go uses substantially less memory than the java version

It’s hard to see from the graph, but the java version peaks at around 470 MB of memory during the load test. The go version doesn’t crack 11; its peak usage is around 10.4 MB.

Soooo, I don’t really care about the difference in performance here. Both apps handle this request volume just fine, and the differences in latency are negligible when you add back in the RTT between the client and server. But come on, the java version uses 45x more memory than the go version does. Ignoring CPU, I could run something like 1400 hashbash-web pods on an m5.xlarge ec2 instance if I were deploying the go version. I’d only get about 25 java pods on an equivalent instance.

This brings me to my conclusion. I would be lying if I said the java version of these applications wasn’t easier to write than the go. With spring boot, I got much of my prometheus integration, http request parameter deserialization, database schema management, and rabbitmq topology declaration and error handling for free, just by adding a few pom dependencies. I had to write a fair bit of code in places to accomplish the same in go. That being said, the difference in resource utilization is clearly substantial between java and go, and one has to make an evaluation when starting a project if the out-of-the-box features of a spring boot-type framework are really worth it.

Ultimately hashbash is exactly the kind of small application for which I think go is an excellent choice. Knowing both languages fairly well, I think the two versions of this project are comparably maintainable, and I wouldn’t be unhappy owning and adding features to either version. With that in mind, I think the tremendous resource savings that can be reaped using go instead of java make it the best choice here.

Can I rewrite everything in go now?