Using RabbitMQ With Rails to Communicate Different Microservices

RabbitMQ is an open source message broker, and it is used to create messaging applications. This is very useful if you want to communicate two or more microservices asynchronously, because it decouples the message sender from the consumer, and also improves availability, since RabbitMQ can keep messages in a queue indefinitely until a consumer processes them. So the objective of this post is to take advantage of these benefits, communicating different services on Rails or other frameworks/languages.

But first, a brief explanation of RabbitMQ and some basic concepts.

Advance Message Queue Protocol in a Nutshell

RabbitMQ uses the Advance Message Queuing Protocol (AMQP), but it also supports others. This is a programmable protocol, so entities and routing schemes can be defined by applications themselves, and not necessarily by a broker administrator.

The model and data flow of this protocol, has the current structure:

• Messages are published to exchanges.
• Exchanges then distribute message copies to queues using rules called bindings.
• The broker either delivers messages to consumers subscribed to queues, or consumers fetch/pull messages from queues on demand.

Who creates these entities?

When a publisher creates a new message, the exchanges must be created; and the same for the consumers; the queues must exist to subscribe to one of them. But as AMQP is a programmable protocol, either the publishers or the consumers may create them before publish or subscribe to. But then, what is the best option?

If our objective is to decouple publishers and consumers, and also improve availability as we said in the beginning, the answer may not be that simple. If the publisher creates the exchanges, as well as the queues and the bindings between them, then your services are not fully decoupled, since the publisher then must know all the queues that may be binded to the exchanges. On the other hand, if the consumers create everything, the problem persists, as it must know all the exchanges that the complete system needs to work, beforehand.

In both cases, if some queue name change, or some exchange needs to change, you may need to redeploy all the consumers or all the subscribers… that’s not a fully decoupled microservices architecture.

Another option is that consumers only create the queues and the bindings, and the publishers create the exchanges. This solves our coupling issue, but the availability may be affected. If the publishers are deployed first, then the exchanges will not have a binding to the queues, so if a message is generated then it will, therefore, be lost. This issue may be controlled if you deploy services in the correct order, but it is not the ideal if you have a complex microservice architecture.

A different approach may be that neither publishers nor consumers create an entity. You can use the RabbitMQ management command line tool to import the configuration of all the broker entities from a JSON file:

rabbitmqadmin -q import rabbit.definitions.json

This approach solves all previous issues, and has the advantage to be very simple to implement in a continuous integration pipeline.

With these definitions clear, we can finally speak about Rails.

Creating Publishers with Rails

Bunny is a ruby RabbitMQ client that is really simple and easy to use. But with Rails, you need some considerations… especially if you’re using puma.

First, it will be useful to create a connection shared across all the code, to avoid opening new connections and channels with each new message. So you can create a singleton class to manage the connection and the channels for you. But here is the first consideration; if you’re using puma, it will use more than one thread per worker or process. So you will have problems with this shared connection, because this Bunny object isn’t thread safe.

The solution is simple, you can use the gem connection_pool to handle a safe pool of lazy loaded RabbitMQ connections:

class RabbitConnection
  include Singleton
  attr_reader :connection
  def initialize
    @connection = Bunny.new(#connection options)
    @connection.start
  end
  def channel
    @channel ||= ConnectionPool.new do
      connection.create_channel
    end
  end
end

Then, you can call this instance and send a message to the corresponding exchange (in this example, a direct exchange):

RabbitConnection.instance.channel.with do |channel|
  exchange = channel.direct('exchange.name', no_declare: true)
  exchange.publish('msg', opts)
end

Here, it’s important to consider another subject; remember that we created all the entity structure with RabbitMQ’s management command line tool. So all the queues, exchanges and bindings already exist. For that reason, it’s important to add no_declare option, which skips the creation of the exchange on behalf of the publisher.

Creating Consumers with Rails

For the consumers, you can also use the Bunny gem, but there’s another gem called sneakers, that’s also useful. This gem is a very good option to handle background jobs with RabbitMQ. The problem with sneakers it’s that it also handles the creation of the queue. So, in order to respect the convention that was mentioned before (creating the structure outside publishers and consumers), we must take another path. The other problem with sneakers, it’s that you may already have a background job processor, e.g. sidekiq. Of course you could simply call a sidekiq job inside a sneakers job and let sidekiq handle the background job responsibility, but then, you would be using an entire gem for something that it was not developed for.

For this reason, we develop a gem called, bunny_subscriber. Its goal is not to be a background job processing framework; just a subscriber that connects to other services that use RabbitMQ. For this reason it does not provide a way to enqueue jobs, nor does it have scheduling jobs or retries with configurable times.

Bunny subscribers just use RabbitMQ structure, and if they fail to complete a job, they not send the ack signal to the message broker, so no messages are lost, even if something fails. It also works well with sidekiq, so you can just call a sidekiq worker:

module Subscribers
  class SomeSubscriber
    include BunnySubscriber::Consumer
    subscriber_options queue_name: 'some.queue'
    def process_event(message)
      SomeSidekiqWorker.perform_async(message)
    end
  end
end

Then you can take advantage of all sidekiq background job processing features, and use this gem as a simple consumer of messages.