How jemalloc improved memory usage of our Rails application
Our backend systems at Bloom & Wild are written using Ruby on Rails. We have what people might call a pretty standard Rails stack, with an API and some admin tools, powered by a PostgreSQL database, and we use the REDIS powered Sidekiq framework to deal with background jobs.
The thing about big Ruby on Rails apps is that their memory usage can be far from optimal. And if you’ve ever had to debug a memory leak in a large Rails application you’ll probably know it’s a very difficult process. On top of that, chances are the leak is in one of the many gems you may be using as a dependency, making the fix even harder, as you may not be familiar enough with that code to make the changes upstream.
Since we moved to AWS Fargate last year, we needed to be extra careful with how much memory our applications use:
- Use too much memory and you’re going to have a very sad face when the AWS bill comes at the end of the month.
- Use too little and you risk Fargate killing your containers, resulting in potential downtime while new ones get booted to replaced the killed ones.
We had to be creative in finding a pragmatic solution to this problem ourselves by automatically restarting containers safely when they were using too much memory. It was a good enough fix for the symptoms.
You can see in the below chart what the container percentage of memory used of one of our API containers was over time. The big drops is memory usage are deploys or container restarts.
Meet jemalloc
While trying to understand a bit more why this was happening we bumped into this article from the creator of Sidekiq: Taming Rails memory bloat. The author explains how switching from the standard glibc memory allocator to jemalloc improved the memory usage of their sidekiq worker processes dramatically.
We decided to give it a go in the hope that we would see similar improvements. We followed the instructions found in this Github issue for the official ruby Docker images, and modified our Dockerfile to install and use the necessary libraries:
To our delight, after testing the changes didn’t have any obscure and negative side effects in the correctness of the application, the results were better than we had anticipated. While our application’s memory usage still kept growing over time, it did so at a much slower pace. For comparison, see below a similar chart for our API memory usage after our changes:
And for our Sidekiq workers, the results were even more astonishing. Find below memory usage over time before and after our changes:
Conclusion
Switching to jemalloc as the memory allocator for our Rails application resulted in much better memory usage over time. Since we host our application in AWS Fargate, where you pay an amount that scales linearly with memory usage, this allowed us to reduce our monthly AWS bill significantly since we were able to reduce the memory allocated to our Fargate containers.
The memory improvements were significantly better in our Sidekiq workers, since the glibc allocator memory issues are affecting multithreaded environments, such as Sidekiq workers that make heavy use of concurrency.
