Migrate to Modern Streaming using Starlight for RabbitMQ

This post introduces you to Starlight for RabbitMQ and shows how you can use it to easily migrate your RabbitMQ application to Apache Pulsar for modern streaming.

In a nutshell, Starlight for RabbitMQ allows existing RabbitMQ applications to use Apache Pulsar as the native message processing provider with minimal changes.

Apache Pulsar’s open-source and cloud-native nature allows enterprises to move their RabbitMQ applications to run in any cloud environment, as well as on-premises. Now with Starlight for RabbitMQ, you can tap into the open-source power and horizontal scalability of Apache Pulsar for all your RabbitMQ applications without rewriting them first.

To name some of its main highlights, Starlight for RabbitMQ is:

  • Fully open-sourced.
  • Blazing fast with independent scalability of compute and storage.
  • RabbitMQ-compatible for drop-in replacement.
  • The quickest path to adding message retention and replay capabilities to RabbitMQ applications.
  • A component of a modern messaging strategy that reduces the total cost of ownership (TCO).

We detailed the “what” and “why” of this exciting API in our Starlight announcement. In this blog post we’ll walk you through the thinking behind Starlight for RabbitMQ, and show you how you can easily migrate a RabbitMQ application to Apache Pulsar.

Under the hood of Starlight for RabbitMQ

Depending on the use case, Starlight for RabbitMQ can be deployed in a broker as a protocol handler or can be launched as a standalone java application.

AMQP 0.9.1 (the protocol RabbitMQ uses) employs the concepts of exchanges, queues, and bindings to provide basic routing capabilities inside the message broker. These concepts need to be mapped to Pulsar topics and features.

One important architectural decision is that Starlight for RabbitMQ doesn’t interact directly with the managed ledger like in other RabbitMQ integrations for Pulsar. Interacting with the ledger has the advantage of being performant, but the disadvantage is that the broker (which interacts with the ledger) must have ownership of the topic.

Since in AMQP 0.9.1 there is a many-to-many relationship between exchanges and queues, all exchanges and queues and related topics must be owned by the same broker. There are techniques to do this using topic bundles, but the result is that a full AMQP virtual host can be handled by only one broker at a time, which becomes an issue for scalability.

Instead, Starlight for RabbitMQ acts as a proxy and uses the Pulsar binary protocol to communicate with the brokers. This means it can leverage Pulsar features like load balancing of the topics on the brokers, batching of messages, partitioning of topics, and load balancing of the data on the consumers.

On the publish side, an AMQP exchange is mapped to a topic. Depending on the type of exchange, the publish routing key is also included in the topic name.

Figure 1: Diagram showing how Starlight for RabbitMQ for Apache Pulsar publishes messages.

On the consumer side, Pulsar shared subscriptions are used to represent the AMQP bindings from an exchange to a queue. When creating an AMQP queue consumer, the proxy creates Pulsar consumers for all the bindings/subscriptions of the queue.

When you unbind the queue, the subscription isn’t deleted right away since the consumer can be lagging. Messages will still be received from the subscription and filtered if their position is past the end of the binding. Starlight for RabbitMQ periodically checks if all messages from the closed binding have been acknowledged. Once the messages have been acknowledged, Starlight for RabbitMQ deletes the corresponding subscription.

Figure 2: Diagram showing how Starlight for RabbitMQ for Apache Pulsar consumes messages.

Pulsar shared subscriptions are used to represent bindings. This is because they enable the use of individual acknowledgments, which is a key feature that both RabbitMQ and Pulsar have in common to support message queue use cases.

Consistent metadata store

Starlight for RabbitMQ uses Apache Zookeeper to store the AMQP entities metadata consistently. The existing ZooKeeper configuration store can be reused for this, and Starlight for RabbitMQ will employ the /pulsar-rabbitmq-gw prefix to write its entries in ZooKeeper.

Security and authentication

Starlight for RabbitMQ supports connections using TLS/mTLS to ensure privacy and security of the communication. It also supports the PLAIN and EXTERNAL mechanisms used by RabbitMQ. Internally, it will use the same AuthenticationService as Pulsar and map these mechanisms to existing Pulsar authentication modes.

PLAIN mechanism

The PLAIN mechanism is mapped to the AuthenticationProviderToken mode of authentication. The username is ignored and the password is used as the JSON Web Token (JWT).

EXTERNAL mechanism

The EXTERNAL mechanism is mapped to the AuthenticationProviderTls mode of authentication. This is the equivalent of the rabbitmq-auth-mechanism-ssl plugin.

Starlight for RabbitMQ can connect to brokers that have TLS and/or authentication, and/or authorization enabled. To perform its operations, Starlight for RabbitMQ proxy currently needs to use an “admin role”. Future versions will relay the principal authenticated to the proxy and use a “proxy role” so operations on the broker will have permissions from the originating application.

At the moment there’s no authorization built in the proxy, but this is on the roadmap.

Multiple proxies support

Multiple proxies can be launched at the same time for scalability and high availability needs. The proxies are stateless and can be started and stopped at will. They share their configuration in Zookeeper so you can create/delete/bind/unbind exchanges and queues on any proxy, and it will be replicated on the other proxies.

Publishing messages can be done on any proxy. On the receiving side, messages will be dispatched evenly to all connected AMQP consumers since the Pulsar subscriptions are shared.

Figure 3: Diagram showing how Starlight for RabbitMQ for Apache Pulsar handles proxy clustering.

Note that the current version of Starlight for RabbitMQ only supports authentication as a security mechanism. Future versions will implement authorization with access controls on the vhosts/exchanges/queues on one side. On the other side, it’ll relay the original principal to also implement this on the Pulsar topics.

Lastly, the current version supports the “direct” and “fanout” types of exchanges. In future releases, we intend to add the “topic” and “headers” types of exchanges.

Start streaming with Starlight for RabbitMQ

Starlight for RabbitMQ is now included in DataStax’s Luna Streaming Enterprise support for Apache Pulsar. You can also dig into the Starlight for RabbitMQ source code on GitHub (available under the Apache license) to learn more about how you can easily leverage the power of Apache Pulsar for your RabbitMQ applications.

Follow DataStax on Medium to keep up with our latest announcements and developer resources on all things Apache, Cassandra, streaming and more. To join our growing developer community, follow DataStaxDevs on Twitter.

Resources

  1. GitHub repo for Starlight for RabbitMQ for Apache Pulsar
  2. Certificate Authentication Mechanism for RabbitMQ
  3. DataStax Luna Streaming
  4. Apache Pulsar
  5. RabbitMQ

--

--

--

We’re huge believers in modern, cloud native technologies like Kubernetes; we are making Cassandra ready for millions of developers through simple APIs; and we are committed to delivering the industry’s first and only open, multi-cloud serverless database: DataStax Astra DB.

Recommended from Medium

How Much Flutter Developers Earn In Europe in 2022

The Magic of Platform-as-a-Service: Months to Weeks

A child in a pink jumper that reads “dreaming” stacks wooden blocks on top of each other on a white desk.

What’s new with rdiff-backup?

How to Set up Local Hadoop Cluster with Oozie

DIVE — Decentralised Identity storage & verification protocol

#LearnWeb3

Firebase Database — how to survive billing issues when you are a baby in the woods

Suggestions on Creating a Product Strategy for Existing Software

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
DataStax

DataStax

DataStax is the company behind the massively scalable, highly available, cloud-native NoSQL data platform built on Apache Cassandra®.

More from Medium

Stargate gRPC: The Better Way to CQL

Supercharge Your Lambda Deployments With Kubernetes

Kafka vs RedPanda Benchmark (also Tarantool and Clickhouse as queue)

How to set up a usable distributed sqlite DB using a microk8 multi-architecture cluster