Geek Culture
Published in

Geek Culture

RabbitMQ Shovel

“Such a shovel, it seemed a waste not to use it.”
Daniel Kraus, Rotters

Photo by Andrey Metelev on Unsplash

In the message broker RabbitMQ, a typical setup is to deploy a single RMQ instance that resides in a box. The job of RMQ is to deal with incoming messages and make sure these are forwarded to the correct destination.

The publisher sends the messages to an exchange which are the message routing agents. The exchange routes the messages based on routing keys to different queues. Each queue has consumers attached who will consume the messages and process them.

RabbitMQ Broker

Now, what if we need a queue-based system where we have multiple queues in a single broker, and each RMQ instance can send and receive messages to one another. A second use case could be offloading messages from one queue to another in a different RMQ broker. That's where Shovel comes into the play.

R️abbitMQ Shovel continually moves messages from a source to a destination. The source and destination can be situated in the same cluster with different virtual hosts or in different geographic and administrative domains. It can move messages between queues and exchanges within the same broker or it can act as a bridge between two different brokers.

A shovel can be used to balance the load of a queue or when we need to take messages out of one RabbitMQ broker and insert them into another. In essence, the shovel connects to the source and destination broker, consumes messages from the queue, and re-publishes each message to the destination broker.

RabbitMQ Shovel

Why do we need Shovel

  • Loose coupling
    A shovel can move messages between different RMQ clusters that may have different users, RMQ version, vhosts, and messaging protocols (AMQP 0.9.1 and AMQP 1.0).
  • WAN Friendly
    Shovel uses AMQP client connection under the hood. It is designed to tolerate intermittent connectivity without message loss by performing Consumer Acknowledgements.
  • Configurable parameters
    Shovels are highly tailorable and can be configured to perform any number of explicit methods.

Types of Shovels

There are two types of shovels- Static shovel and Dynamic shovel.

Static Shovel is defined in the configuration file and the node restart is required to apply any update. This is more general in nature as queues, exchanges, and bindings need to be declared manually at startup. If multiple RMQ nodes are present in the cluster, then a static shovel should be defined for all the nodes in the config file. Each shovel will only start on one node, and if the node goes down, it will move to another healthy node.

Dynamic Shovels are defined in runtime parameters. They can be created and deleted at any time and don't require a node restart. These are more automation-friendly as the queues and exchanges used by shovel are declared automatically. In a dynamic shovel, we don't need to manually define it for all the nodes. This is done automatically and each shovel will only start on one node and failover to another when needed.

Set up a shovel

Enable Shovel plugin

Installing a shovel is plain sailing. This can be done by using RMQ CLI.

rabbitmq-plugins enable rabbitmq_shovel 
rabbitmq-plugins enable rabbitmq_shovel_management

Shovel mapping

Shovel mapping can be created from the RMQ Management dashboard. Go to the Admin section and click on Shovel Management.

Syntax of the RMQ URI is amqp://username:password@host:port/vhost

RabbitMQ — Shovel Management

Shovel params:

  1. Virtual Host: Select vhost from the dropdown.
  2. Name of the shovel.
  3. Source: Enter the RMQ URI and source queue/exchange.
  4. Destination: RMQ URI of destination instance and queue/exchange.

Once all the details are in place, click on Add shovel button.

The status of the shovel can be visible from the Admin > Shovel Status tab. Status value can be:

  • Starting: Shovel is starting and creating resources and trying to connect to endpoints.
  • Running: Shovel is successfully connected to both endpoints and consuming from source and republishing to destination.
  • Terminating: Shovel has terminated or it ran into some exception. The reason is visible in the logs.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store