Creating sneakers workers dynamically

Sneakers (RabbitMQ) for Cheetah

Prolog

At Cheetah Technologies we fulfill deliveries from different locations (warehouses) spread across the US. We integrate with ERP and WMS systems to get live inventory and pricing updates. Our CatalogService connects with the ERP system to get catalog related data including packaging variations, images, descriptions, etc.

Technology

Most of our backend is Ruby-based applications so after a short research we decided to go with RabbitMQ + Sneakers for communication between services. Both bunny and sneakers are very mature gems in our community so the decision was easy.

The desired queues structure should be as follow:

# defining the queue segments
queue = "#{type}.#{city}.#{location}"

While implementing topic exchange this structure enables us to route messages using some fancy patterns.

That said currently we use topic exchange with a single binding as a queue name which is a direct exchange implementation. The nature of a programmer is to always strive for a generic approach and moving from already implemented direct exchange to topic is much harder then just create those fancy bindings in the future.. if we would ever want to implement wildcards.

The challenge

Sneakers workers consume messages from a queue defined on a class (worker) level:

This means that if we want to create queues like:

inventory.san_fransisco.wh_2
inventory.san_fransisco.wh_3
inventory.san_jose.wh_1
inventory.los_angeles.wh_1

We’d have to create four workers.

Having multiple types of messages and multiple locations permutations meaning creating a lot of files in our project.

Metaprogramming-fun-fun-fun

Let’s create a module that will:
1. Include the Sneakers::Worker module
2. Pass queue options to the from_queue method
3. Define the work method implementation

Now we need to create the worker dynamically including the workable module in it.

And to wrap this up we’ll need to iterate on available locations to build the queues and workers names.

The queue could be something like inventory.san_fransisco.wh1 and the created in memory worker class would be InventorySanFransiscoWh1Worker.

This would be yielded as a message_handler

InventoryService.call(message)

So all that is left to do here is calling the InventoryWorkers when sneakers server starts.

# initializers/sneakers.rb
if Sneakers.server?
  SneakersBuilder::InventoryWorkers.call
end

Epilog

We were able to create as many classes as we need dynamically without the need to maintain physical files in our project. Of course, if the number of required workers would be much smaller we would prefer to go with the regular approach as meta has it’s dark sides.

So far we have 4 types of builders where each defines a subset of workers based on locations or channels defined in yml files. This way each new entry will require a restart and new workers will load into memory.

We love Ruby.