Fighting the Heroku `Error R14 (Memory quota exceeded)` issue.

Douglas Liu
Aug 14, 2018 · 5 min read
Photo by CloudVisual on Unsplash

I was trying to implement a feature for my play-scala application running on Heroku, which would require serving a large json result (around 2.5mb) to client. Not before long, I saw the R14 error being displayed in the logs.

Error R14 (Memory quota exceeded)

Oh, I know Heroku free tier has only 512mb, but I seriously think it’s something I need to address.


Know Your Enemy

It only makes sense to put efforts on the correct direction. We need to know what’s causing the issue first.

First, use the below to enable log metrics.

heroku labs:enable log-runtime-metrics
Enabling log-runtime-metrics for myapp... done
$ heroku restart

The log metrics will constantly print log message like the below.

2018-08-10T08:20:12.324796+00:00 heroku[web.1]: source=web.1 dyno=heroku.xxxxxxxx sample#memory_total=491.32MB sample#memory_rss=491.20MB sample#memory_cache=0.12MB sample#memory_swap=0.00MB sample#memory_pgpgin=176221pages sample#memory_pgpgout=50442pages sample#memory_quota=512.00MB

Among all, I’m more concerned about the memory_total. So long as the number exceeds memory_quota, R14 will be raised and your application will start using swap.

Wait, isn’t Heroku configured default mx (maximum heap size) to be 300mb ? How is this possible my application used more than 512mb of memory ?

Well, heap memory isn’t everything. There’re metaspace and the other non heap memory used by JVM. Take metaspace for example, this was introduced in Java 8, and the default is your virtual memory. If you don’t set a limit, it may grow beyond what physical memory you have.

The above does’t provide much insights, more like a confirmation of the truth. We shall continue on our discovery. Let’s install Heroku’s java debug plugin.

heroku plugins:install heroku-cli-java

This give you access to the below command

heroku java:jconsole
heroku java:visualvm
heroku java:jmap
heroku java:jstack

jconsole provides good visual overview on what’s been used. Take a good look at the loaded classes. If the number doesn’t stop growing, you may be having class memory leak that could possibly be caused by dynamic proxy. Set something like to make sure it doesn’t grow too high.

My observation shows the number of loaded classes is rather stable after multiple loads of the problematic API. I’m leaving the default for the moment.

jmap is also very handy as it can be used to get the detailed heap usage, so that you know, which class uses the most memory. It can also used to extract heap dump and download it to local. You may later analyzed it with eclipse MAT.

It does tell me something very important.


Fix the ForkJoinPool size

In my case, the first issue that pops up is, I’ve got tons of ForkJoinTask created. It’s so many that it ranked first on the jmap class list by bytes. Being reactive means we’d need to create a lot of Futures in the codes. But probably it becomes too many. Something was wrong.

It turns out I’ve used my testing parameter for deploy to Heroku.

// This is my previous configuration. Too high for Heroku.fork-join-executor {
parallelism-factor = 8.0
parallelism-max = 200
}

My dyno runs on a virtual machine with 8 cores. The above means, my pool is created with 64 threads, and may grow up to 200 threads. Given I’ve more than one pool, This is certainly too high. No wonder jconsole show my active thread to be 70+. After some tuning, it was now sub 40 on average.

// This is my new configuration. Looks good to me.fork-join-executor {
parallelism-factor = 3.0
parallelism-max = 64
}

Slick numThreads

I also found I’ve got 20 slick threads, which means slick may use 20 connections simultaneously. This is the default, but may not be very realistic on Heroku, especially when you’re on free tier + free postgresql, where 20 is the connection limit. I bet you’ll want to keep a few for your database tool. Consult the hikariCp wiki for guidance on how to configure the pool size.

Once you decided the size, configure the numThreads in your application.conf. I choose to use 16.

slick {
dbs.default {
driver = "slick.jdbc.PostgresProfile$"
...
db.numThreads = 16
}
}

Serialize large Json array

The dominating issue is, of course, loading the large json objects and serialize them in memory. This is hardly something configuration can help. We’ll have to use streaming API.

Since Play 2.6, the framework started to support Akka Http. Akka Http comes with a nice feature, akka-streams. I’m rewriting my API to use this. It’s not a big deal, but I guess it will have to become a second story.

Use what you’ve got

I’ve tuned quite a few so far, but I’ve noticed that the JVM is reluctant to perform a full GC even when nobody is using the application. If I’ve got a R14, the usage wasn’t reduced overnight. This leads me to think, does the JVM really knows how much memory we’ve got ?

I re-read the Heroku JVM tuning article and realize, the JVM might be mis-lead because we’re running in a container. When you use to run , you’ll find the host has big memory, not just 512mb. JVM must be confused about it.

You need to add the below to your if you’re on JDK 8

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

For JDK 9, use the below

-XX:+UseContainerSupport

As a matter fact, this is the first thing you should do. I’ve no idea why I skipped this. To be honest, I think this should be the default options for ALL java application on Heroku.


I have also tried the below, which may be helpful to improve the overall memory usage. But they lack the theory to support them, so they’re listed here only for references.

Use G1 garbage collector

In my previous project, I found the G1 garbage collector very efficient, it rarely create pauses that’re noticeable. So I opt to use it. This is done by configuring JAVA_TOOL_OPTIONS in Heroku Config Vars. I’ve also added options to print GC details, but I found them too verbose and didn’t provide much help. I’ll turn them off very quickly.

-XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+UnlockDiagnosticVMOptions

In the end the JAVA_TOOL_OPTIONS is the below, as they provide the critical information I’d like to see at current stage.

-XX:+UseG1GC -verbose:gc

Full GC on application starts ?

This is irrelevant, I believe a Full GC on application starts is only a false assumption of mine trying to force the JVM to perform the full GC. It doesn’t help in the long run.


After a while of testing, I found a manual full GC right after application start seems to be helpful. It doesn’t seem to create any issue anyway, but have significantly reduced the memory usage. The total_memory have dropped to around 400mb, even when I invoked the problematic API 10+ times.

However, it could also be the result of the adoption of streaming API.

Did you learn something new? If so please:

clap 👏 button below️ so more people can see this

Sohoffice

Full Stack Consultancy and Development. Worked from Taiwan, Serviced the World.

Douglas Liu

Written by

Problem solver. Found love in Scala, Java, Angular and more …

Sohoffice

Sohoffice

Full Stack Consultancy and Development. Worked from Taiwan, Serviced the World.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade