Borrowed from NewRelic Blog for RailsConf 2013, pretty isn’t it?

Threading with Rails

The team here at Points Travel love Ruby on Rails. It’s been our stack of choice since 2012 and is battle-tested for everything we can throw at it from multi-tenancy to multi-threaded searches.

Today, we released a version of our app (https://pointshound.com) that finally addressed a long standing issue with Rails that consumed a massive number of database connections even though most were idle.

I/O Processing in Parallel (gem)

When most Rails developer think about handling multiple things at once, the first thought is to use Sidekiq or Resque and if you’re truly die-hard, JRuby. Well, the first question you should ask is what exactly is the thing you’re doing concurrently (see Quora thread about Parallelism vs. Concurrency if you’re hung up about terminology).

In our case, our app searches multiple hotel suppliers then de-dupes them, prices them, and presents them to our customer. As with most 3rd party systems, the response time can vary and you’re doing a lot of waiting for a response. The processing of the response in JSON or XML can consume CPU but the majority of the time is I/O wait.

Thankfully, there’s Puma, which excels at allowing blocking IO to be run concurrently with the MRI Global Interpreter Lock (GIL). To keep things simple (and back in the day, we were running on Heroku with no cheap Redis plans) we explored the parallel gem by grosser. Running threads concurrently was as simple as this:

# 3 Threads -> finished after 1 run
results = Parallel.map(['a','b','c'], in_threads: 3) {|letter| ... }

No Redis, no queues, no background job required, just a simple block of code run in a new Thread. Even better, it doesn’t require extra memory since memory is copied on write. Yes, it can be that simple and wonderful (please star the repos!).

A Few Gotchas

In development mode, Rails does it’s magic class autoloading. When you create a new Thread, it doesn’t get bestowed any powers of dynamic class loading, so any class not already instantiated before you start processing in your Thread will make your Rails app mysteriously halt. This became evident when we refactored code blocks into new Command-pattern classes and Rails would just stop. To get around this, you can instantiate any dependent class in your class initializer or keep all the code in one class.

In addition, ActiveRecord has it’s own ideas on when to grab database connections and how to manage them magically for you. To avoid running out of connections, you have to manually manage the connection for the created thread by doing this:

Parallel.map(inventory_map.values, :in_threads => n) do |inventory|
    ActiveRecord::Base.connection_pool.with_connection do
        # search hotel inventory supplier source
    end
end

This worked great and kept the connections clean and tied to each specific thread until we introduced the apartment gem. It’s probably the best multi-tenancy gem with its ease of integration using PostgreSQL schemas to separate tenants, however, with great power comes great responsibility.

Newly spawned threads aren’t aware of which tenant context it’s in, so you have to initialize the thread with Tenant.switch!(tenant_name). This innocuous command ALWAYS uses a database connection before any SQL request is actually made. So, let’s say you have four hotel suppliers running in four separate threads, you now have four new database connections plus the one you already have from the original Rails request. This quickly gets out of hand if each customer on your site requires five database connections per search.

ActiveRecord Implicit Connections

We were now seeing 50–60 connections per application server with 90% of the connections idle. Memory usage was getting gobbled up on our database server. We discovered setting prepared_statements: false in database.yml helped stave off the Linux OutOfMemory killer, but we still noticed heavy memory usage. We thought about using PgBouncer, a wonderful, lightweight connection pooler but it had a few caveats and each customer would still require five connections for the duration of the request.

So, we decided the best way to reduce the number of database connections was to NOT use database connections. Duh. Ok, let me explain. We identified the smallest unit of work to handle the hotel supplier request/response into the thread body, and then made sure anything from the database was was cached or passed into the thread. The hard part was that ActiveRecord makes a database connection implicitly, hiding where the database may actually be used.

There was this nice monkey patch for this exact problem (Internet for the win!), but it didn’t work with Rails5 so we tweaked it a bit. Note, we set the tenant name onto the thread since calling Tenant.current_name requires a database connection and we need to know which tenant we’re in for I18n translation (a post to itself) to work properly.

Parallel.map(inventory_map.values, :in_threads => n) do |inventory|
    ActiveRecord::Base.connection_pool.with_connection do
        Thread.current[:tenant] = tenant_name 
        ActiveRecord::Base.forbid_implicit_checkout_for_thread!
        ...
    end
end

Proof in the Pudding

Using our NewRelic monitoring (unfortunately they are deprecating the Servers tab in May 2018 — possibly another Medium post about us moving towards Prometheus), we can see the memory usage of our database plummets to a comical percentage. Just before we released, the Google Cloud Console was recommending to increase the instance size because it was being over-utilized!

In Conclusion

With database connections under control, we are now able to further parallelize hotel searches by breaking out each hotel supplier into multiple threads, so a customer search may have anywhere from five to fifty-five threads running with only one database connection.