OverAttired — Background Processes

One of the unique features of our web app is its background processing. Whenever a new user signs up for OverAttired, they get a welcome email automatically with a personalized products available at the OverAttired store.

At the same time, we have another worker in the background that scrapes their Etsy store every morning at 9 AM to for new products or products that have recently sold — and if a product fits the measurement criteria of specific users, the user will get an email.

WORKER & BACKGROUND PROCESSES

What is Sidekiq? Sidekiq is one of the many options for converting long-running jobs into a background process. Sidekiq is known for saving memory usage because it handles multiple jobs concurrently “using threads instead of processes” (whatever that means as I’m just reading this off of RailsCast).

Where to begin? To use Sidekiq, we needed to establish a Worker class. We created a new directory called overattired/app/workers, which our application will know to load automatically. For every files in this folder, you’ll notice that we included a Sidekiq::Worker module as well as a Sidetiq::Schedulable module module for automatizing Sidekiq.

# overattired/app/workers/scraping_worker.rb
class ScrapingWorker
include Sidekiq::Worker
include Sidetiq::Schedulable
.
.
.
end

The perform method contains code that we want running in the background. When we are calling ScrapingWorker.perform_async, our application will make running this method as a “job” and add this “job” to Redis to be performed asynchronously. You can view all the jobs in the queue in http://0.0.0.0:3000/sidekiq/scheduled.

How does our application run this method automatically?

We downloaded a gem called Sidetiq and called this module under our ScrapingWorker, which let us put this one line of code to make this worker perform automatically while redis, sidekiq, and local servers are running:

recurrence { daily.hour_of_day(16) }

To test whether Sidetiq was working, we changed the recurrence to run every 10 minutes:

recurrence { hourly.minute_of_hour(0, 10, 20, 30, 40, 50) }

Troubleshooting Step-by-Step

  1. Look for the “perform” method
# overattired/app/workers/scraping_worker.rb
def perform
store_data_from_etsy
# see below
mail_users
end

2. This method calls on two other methods: “store_data_from_etsy” and “mail_user”

The “store_data_from_etsy” method calls on the Etsy API then parses the incoming store data to look for any products that have sold and any new products that have been added to the Etsy store. It will then update these findings in the database.

# overattired/lib/etsy.rb
def store_data_from_etsy
all_active_listings = []
offset_array = ["offset=0", "offset=100"]
offset_array.each do |offset| .
.
.
.
end

After Etsy has been scraped, the “mail_users” method is programmed to automatically run the “get_new_matches” method on all the non-admin users. It will send out emails to all non-admin users that received new product matches.

To prevent faulty data, we’ve also weeded out users that have no products matches in history (folks with unique body measurements) and users who have not entered in measurements at all.

# overattired/app/workers/scraping_worker.rb
def mail_users
all_users = User.where(:admin => false)
all_users.each do |user|
new_matches = user.get_new_matches
if ( user.measurement == nil ) || (user.match == {}) ||
(new_matches = {})
else
MatchMailer.match_email(user, new_matches).deliver_now
end
end
end

3. The “mail_users” method calls on the “get_new_matches” method

The “get_new_matches” method will return an array of products that fit a user’s measurements that was recently added to the Etsy store.

# overattired/app/models/user.rb
def get_new_matches
all_product_matches_array = self.match_hash_flatten
new_product_matches_array = []
   self.matches.each do |match|
product = Product.find(match.product_id)
if !all_product_matches_array.include?(product)
new_product_matches_array.push(product)
end
end
   new_product_matches_array.each do |product|
Match.create(product_id: product.id, user_id: self.id, emailed:
true, emailed_date_time: DateTime.now)
end
   match_array_to_hash(new_product_matches_array)
end

It will call on the “match_hash_flatten” method, which creates an array containing all products that fit the user’s measurements, including the new ones that were just scrapped from Etsy and the ones that are sold.

# overattired/app/models/user.rb
def match_hash_flatten
all_matches_hash = self.match
all_matches_array = all_matches_hash.values.flatten!
return all_matches_array
end

It will also call on the “matches” method, which was a method automatically created by the Rails app when we created the Match model. Calling “matches” on a user will return an array of matches (linking products to users prior to the recent Etsy scraping).

# overattired/app/models/match.rb
class Match < ActiveRecord::Base
belongs_to :user
belongs_to :product
validates :user_id, :uniqueness => { :scope => :product_id }
end

The “get_new_matches” method returns an array of all new products that just came in from the recent Etsy scrape by comparing old and new products.

Show your support

Clapping shows how much you appreciated Regii’s story.