delight.fm: day 13

Working with jobs

I’m retrieving artist and album data successfully, but it’s taking a while to process. The tracks in a Spotify library are retrieved in batches and then processed to extract the other parts (artists/albums). Works great. The problem is that it takes a while.

It was clear when I was testing locally that the experience of waiting for all this data to be compiled when you log in was not great. And when I deployed for testing, it was causing errors.

So I split the work of retrieving that data into a job that runs in the background. I started with this:

class ImportSpotifyTracksJob < ApplicationJob
  queue_as :default
  def perform(user, offset)
    Track.import_spotify_tracks user,
      user.spotify_tracks(limit: 50, offset: offset)
  end
end
class Track
  def self.update_user(user)
    offset = 0
    while offset < count
      ImportSpotifyTracksJob.perform_later user, offset
      offset += 50
    end
  end
end
Track.update_user(user)

To retrieve one user’s tracks, you have to make many calls to the API, so I’m looping through the sets of records and processing each set. Each job takes at least 10 seconds, so you end up with many jobs all running in parallel. Unfortunately, they all call the API in rapid succession and you start hitting the rate-limit.

So I moved to more of a recursive approach to delay the next call until the first one has finished. (Recursion is where a function calls itself. I’m sure one of my professors somewhere is very proud.)

def perform(user, offset, track_count)
  Track.import_spotify_tracks user,
    user.spotify_tracks(limit: 50, offset: offset)
  new_offset = offset + 50
  if new_offset < track_count
    ImportSpotifyTracksJob.set(wait: 10.seconds).perform_later(
      user, new_offset, track_count
    )
  end
end

This is working for the moment, but there are still a lot of issues with it. As soon as more than one user signs up at the same time, we’ll probably be right back to hitting the rate-limit. That might be okay or it might not, as long as the jobs rerun. I’ll do some more testing and see.

When you have a lot of work that needs to be done, you should probably consider breaking them up into jobs to do them in the background.