Dynos spinning other dynos with Heroku

Our client base is thankfully getting larger by the day at Ivy and internal reports that we generate off our Postgres database have started maxing out the memory on our poor single worker dyno.

To solve this, we decided to have our worker dyno spin an additional Heroku performance dyno temporarily for memory-intensive, shorter tasks. One-off dynos could have been a solution here, but they are limited in memory. Dynamically scaling a single dyno up and down turned out to be a simple, cost-effective solution.

Adding dynos to the Procfile

We previously had a single worker dyno running all of our background jobs:

# Procfile
web: ...
worker: bundle exec sidekiq -q small_things -q other_things -q reports

Now we’ve added a new worker dyno, and split the queues between the two:

# Procfile
web: ...
julia: bundle exec sidekiq -q small_things -q other_things
winston: bundle exec sidekiq -q reports

You can name your worker dynos whatever you want (although web has to be called web) so I’ve named our two worker dynos after my two favourite workers — Winston and his fellow party member Julia, from 1984.

Once this Procfile is deployed, you’ll see in your Heroku dashboard that your workers have been renamed, and that the additional worker has zero dynos scaled. That is what we want to scale dynamically.

Jobs that start other Jobs

We previously had a single ActiveJob task that was initiated by a reporting panel in our ActiveAdmin interface.

We now have a new ActiveJob task that runs in a reports queue, the only queue run by our new winston dyno. The original task that runs in the default queue spins a new dyno, and at the same time sends a report generation job to the reports queue, to be run as soon as the new dyno loads.

First I created two utility methods that will be responsible for spinning and removing our reporter dyno. I specify the exact dyno (from our Procfile) that I want to scale — winston — and the new quantity of dynos that I want using the Heroku formation API endpoint (https://devcenter.heroku.com/articles/platform-api-reference#formation-update):

# lib/heroku_dyno_handler.rb
# (This needs to be added to your application.rb autoload paths.)
module HerokuDynoHandler
def self.spin_reports_dyno
heroku = PlatformAPI.connect_oauth(ENV['SPIN_DYNO_KEY'])
heroku.formation.update(ENV['HEROKU_APP_NAME'], 'winston', {quantity: 1})
end
def self.remove_reports_dyno
heroku = PlatformAPI.connect_oauth(ENV['SPIN_DYNO_KEY'])
heroku.formation.update(ENV['HEROKU_APP_NAME'], 'winston', {quantity: 0})
end
end

To use the PlatformAPI, add the Heroku Platform-API gem to Gemfile and bundle install. You will also need to authenticate using an API token, which you can get from your Heroku account settings. Notice that I stuck it in an environment variable called SPIN_DYNO_KEY.

# Gemfile
gem 'platform-api'

Now our first ActiveJob task will spin a dyno and then send the second job into the queue:

# ActiveAdmin calls our ReportGeneratorJob
ReportGeneratorJob.perform_later(AutomatedReport::NAMES[:great_big_list_of_stuff])

# Julia receives the task, spins Winston, and sends a new task into Winston's reports queue
class ReportGeneratorJob < ActiveJob::Base
queue_as :default
  def perform(report_name)
HerokuDynoHandler.spin_reports_dyno
ReportRunnerJob.perform_later(report_name)
end
end

# Winston wakes up and runs the reports queue
class ReportRunnerJob < ActiveJob::Base
queue_as :reports
  def perform(report_name)
AutomatedReport.generate(report_name)
HerokuDynoHandler.remove_reports_dyno
end
end

This works surprisingly smoothly, is incredibly cost-efficient (since you end up only paying for the time it takes to run the specific task), and has the added benefit of a sci-fi feel when you watch the logs and witness multiple server instances controlling each other and a dyno committing suicide.