Running a High Traffic Rails App on Heroku’s Performance Dynos
TLDR; If you’re currently using 10+ 2x dynos. Consider using performance dynos. If 20–30 2X dynos, definitely use performance dynos.
Heroku recently announced their new Performance-M ($250/month) and Performance-L ($500/month) dynos. This was perfect timing for us at Product Hunt. Having just launched LIVE, our traffic was out of control.
Here are the new dyno stats
I was excited to see the Performance-L with 14gb of memory and it seemed perfect for us at Product Hunt.
When looking at the PL dyno, the first factor I looked at was cost per Ruby process. For Product Hunt, our current cost was ~$25 per process on a 2X, and based on some napkin math would be ~$17.86 on a PL.
Why look at $ per process?
When scaling a Rails app, one of the key metrics we look at is the # of concurrent requests it can handle. Ruby on Rails applications generally use a lot of memory and are not CPU constrained. When using Ruby MRI (single threaded), this means that a single server (dyno) can only respond to as many requests concurrently as it has Ruby processes running on the machine.
If we’re using Puma and each Puma process uses 300MB of RAM, then a 2X dyno can only run 3 processes (handle 3 requests concurrently).
Results: Product Hunt on the Performance-L Dyno
I already knew that switching to a PL would make sense from a cost perspective. And I knew there would be some performance boost (dedicated CPU, less variable performance). But I wasn’t expecting the insane performance increase we experienced.
Our switch to the Performance-L dyno resulted in a 3X performance increase over the 2X dyno. 😎
Checkout this New Relic response time chart during the transition.
On the left we were running 30 2X dynos. On the right, we are running 3 PL dynos. Same cost, 3X better performance and less variability in request response times.
Note: Your experience will vary. Our large performance increase is mostly due to the more powerful CPU. In our application we pre-render React with rubyracer (this is largely CPU constrained).
Our friends over at Codeship also made the change and noticed similar results.
How to transition to the Performance-L Dyno
1. First, you need to determine how much memory your current dynos are using in production.
You can do this in several ways:
- enable log-runtime-metrics and look at your logs
- use the Heroku dashboard
- Setup a tool like Librato
Divide the total memory usage by the # of processes you have running on a dyno to determine your per process memory usage.
2. Next, in your staging environment, you’ll want to resize your web dynos to Performance-L.
$ heroku ps:resize web=Performance-L --app your-staging-app3. You need to determine the # of processes to run on the Performance-L dyno. To do this, you’ll take 14GB (memory on a PL) & divide it by your per process memory usage. If our app takes 500MB per process, we could fit 28 processes on a PL dyno.
Note: I like to be conservative initially and use a smaller # of processes than I think can fit on the dyno. Better to be safe than to be R14'ing in production later.
4. Now you’ll need to adjust the number of Ruby processes running on each dyno in your staging environment. This next step will vary depending on the server & environment variables you have set. The command will probably look something like the following. You’ll need to reference your puma or unicorn config file for your application to know exactly what ENV variable to change.
$ heroku config:set WEB_CONCURRENCY=20 --app your-staging-app5. It’s a good idea at this point to do a load test & monitor memory usage / CPU load. I recommend using Siege for a simple load test, and monitor it with Librato (upgrade to a higher plan to get more detailed 5 min metrics).
Note: For details on how to do a load test, there is a chapter in my book on load testing Heroku apps.
6. Once you’ve put your staging application through a load test & feel comfortable about the performance (no R14's/meltdowns). It’s time to do this in production. You can do this without downtime, but need to be careful.
Note: Keep in mind your available database connections (or redis/memcached/etc). If each process uses 1 connection, do some quick math and make sure you won’t run out of connections during the upgrade. You can run $ heroku pg:info to check your current database connections and limit.
7. To make the transition in production, first scale down the # of dynos you’re running & then resize to the PL (there is a limit of 10 PL dynos per application, you can get this limit lifted if you email Heroku).
$ heroku ps:scale web=3 --app your-production-app
$ heroku ps:resize web=Performance-L --app your-production-app
$ heroku config:set WEB_CONCURRENCY=20 --app your-production-app8. You’re done! Now monitor Librato/New Relic. After a few hours you will have a good idea of how much memory your application is using on the PL dyno & can adjust concurrency accordingly.
Upgrading to Performance-L dynos resulted in a huge performance gain for us at Product Hunt. It improved user experience with faster page loads and lowered our Heroku bill. If you’re running an application on Heroku in the range of 10–30 2X dynos, you should look into using the Performance-L.
🌟 You can join to my email list here. I send out an update each time I publish something new.
