Rails + RabbitMQ + ActionCable. How to build realtime microservice. (Part I)

A rabbit after the last commit of the day

Context

For few months I am building a crawler to collect restaurant informations and their dishes. The aims is to have as much restaurants as possible in my DB. Have a huge quantity of data is cool but display them is better.

So I decided to build an app dedicated to display data in real time, and only that. In the context of this article we will name this app Dashboard.

The initial spec is to display charts and datas up to date (means don’t refresh data only once a week/day/hours. We want data updated in real time).

I have several solution to do this:

• Use HTTP request (API):

Consume API endpoints each time I go to the dashboard. Quickest and easiest solution. But not efficient, if I have 20 differents charts to display it will make 20 different API calls. And client need to be refresh.

• Use background job (Sidekiq):

What about us background job to synchronize our datas ? Like run a cron job every ten minutes to fetch data from Crawler DB and populate the Dashboard DB ? Definitely not what I want, what will be the correct recurrency to call our job ? I want a real time updated data, if I run it every ten minutes I take the risk to display fake informations while 10 minutes.

• Use message queue (Advanced Message Queuing Protocol)

This is the solution I choose. Basically AMQP allow that a producer (our Crawler) notify our consumer (Dashboard) that a new data has been created, in a real time. Once a new data is added in Crawler it will push this data into a dedicated queue that will be received by the consumer. Efficient, timely and single responsability compliant.

In this first part let’s just configure our producer, in the second part we will create our consumer (Dashboard). Below the system that we are going to build.

Configure our producer

I assume that our Crawler app already exist. Our goal is to push a message in a queue when we create a new restaurant. I’m going to work with RabbitMQ and the Bunny gem. RabbitMQ is an Open Source message broker/queueing system written in Erlang implementing AMQP.

First I have to install RabbitMQ on my machine, as I am on Mac Os so I run:

brew install rabbitmq

You can test that it works by running the following command

/usr/local/opt/rabbitmq/sbin/rabbitmq-server

and go to the admin interface http://localhost:15672/ . Password and username are « guest » .

Perfect. Now we have to add our producer to our app. We are going to use the Bunny gem, a ruby client for RabbitMQ.

gem 'bunny

Then we can create our Publisher class.

services/publisher.rb
require 'bunny'
class Publisher
  class << self
    def publish(exchange, message = {})
      x = channel.fanout("crawler.#{exchange}")
      x.publish(message.to_json)
    end
    def channel
      @channel ||= connection.create_channel
    end
    def connection
      @connection ||= Bunny.new.tap(&:start)
    end
  end
end

Nothing hard here, we instantiate a Bunny instance to start a connection, then we create a channel and we fanout our queue name to this channel. Then we are able to publish our JSON message.

Just one point about « fanout », according to official RabbitMQ documentation:

The fanout exchange is very simple. As you can probably guess from the name, it just broadcasts all the messages it receives to all the queues it knows.

Then we have to call our service and pass our message. As we want to notify our Consumer each time that a restaurant is created we are going to call our publish in a create callback.

# frozen_string_literal: true
class Restaurant < ApplicationRecord
  after_create { publish_to_dashboard }
  private
  def publish_to_dashboard
    Publisher.publish('restaurants', attributes)
  end
end

Now you can open you console and create a Restaurant, go to your admin UI and if everything is ok you should see an Exchange named « crawler.restaurant ».

Congrats. 🎉 You have just created your first exchange. In a second part we will create our consumer and add queue to connect both applications together.

Part two: https://medium.com/@clementrollon/rails-rabbitmq-actioncable-how-to-build-realtime-microservice-part-2-23c5f15c3fef

Let me know if you are facing some issues.