Interservice communication with Redis and Sidekiq remote workers

As our infrastructure continues to grow (and we continue to extract core services) we’re consistently having to sync data between applications.

One of the most common mistakes we see when a team is initially structuring their application as an SOA is to lean on a single database as the central storage bucket backing all of the services.
 — Tammer Saleh (SOA AntiPatterns Talk)

A message queue allows us to decouple the services, and acts as a great temporary holding place while transferring data between applications.

The Gateway Service is a core component of our infrastructure as its responsible for interacting with all of our external hotel partners and making sure we always have the most up to date inventory.

When a message arrives at the Rails App there is a small (yet not insignificant) amount of work that needs to be done before persisting it to the data-store.

A scheduled job was used to batch import messages into Sidekiq for parallel asynchronous processing. However, as the volume of messages grew, the scheduled job became a bottleneck in our pipeline.

We decided to explore the idea of pushing jobs directly from the Gateway Service directly into the Rails App Sidekiq queue.

A thin layer was created to manage the Redis connection and wrap the interface of the Sidekiq client.

Some feedback on the PR reminded me we need to namespace the Sidekiq queue.

Namespacing allowed us to run multiple workers simultaneously in our local environment — This gave us the ability to test an end-to-end integration in development mode.

It’s worth noting that we’ve now introduced connaissance of name between the two services — The application pushing remote work needs to know the class name of the worker in the remote application.

To help avoid leaking this knowledge throughout the codebase we created objects to wrap the internals of each remote worker, and added validations to help enforce data integrity.

The remote worker setup has allowed us to remove the duplicative message queue, and by doing this we’ve increased the throughput by continually streaming jobs directly into the remote Sidekiq worker queues.


Originally published at engineering.hoteltonight.com.