A shredded rabbit: Tanzu RabbitMQ for critical enterprise-grade data flows
This isn’t just a story, nor is it a guide, but a blend of both. It is more about RabbitMQ’s journey as it evolved from a mere concept to the world’s most utilized enterprise messaging system. This article is the first in a series where we will delve into the technical details and explore what sets RabbitMQ and Tanzu RabbitMQ apart as exceptional solutions.
From idea to project
This isn’t my story to tell, as I was just a student back in the 2000's when the concept of a messaging broker ultimately called “RabbitMQ”, grounded in an “open” messaging standard — AMQP (Advanced Message Queuing Protocol) — was first imagined. My introduction to this tale unfolded at RabbitMQ Summit 2023 in Berlin, where Álvaro Videla, a pivotal (remember this word) contributor to the RabbitMQ project, delivered an engaging talk on its inception. It was the kind of story you wouldn’t want to miss, believe me!
My own chapter with RabbitMQ began in early 2017. At the time, I was with a major French automotive company that was reliant on IBM MQ. Then, amid a significant digital transformation, IoT (Internet of Things) devices emerged as a critical component of the company’s infrastructure, necessitating support for MQTT (Message Queue Telemetry Transport) for some essential systems. The middleware guru at the company (who was not me, but who would become my mentor) attempted to integrate this with IBM MQ, but it was no smooth ride. Or perhaps it was, yet within weeks, whispers of a new broker being trialed started circulating: RabbitMQ. “Yeah, right,” I scoffed, “A system named after a furry animal running our production systems? As if…” But oh, how mistaken I was.
RabbitMQ didn’t just prove its mettle by being exceptionally adaptable, scalable, mature, and robust — it was also open-source. That meant access to a vast community brimming with tools, ideas, and implementations. And it changed everything.
From project to product
RabbitMQ’s origin story, as per Wikipedia, begins with its development by Rabbit Technologies Ltd. It’s been on quite the journey since, passing through various extremely talented hands. Acquired by SpringSource (a VMware company) in 2010, our fuzzy-named broker was then snapped up by Pivotal Software in 2013 — born from the spin-out of EMC Corporation and VMware. Pivotal would later be the powerhouse behind revolutionary data solutions such as Greenplum, Gemfire, and the Spring Framework, not to mention our protagonist, the Rabbit.
This is the point where I personally believe RabbitMQ truly began to mature into a system robust enough for the heavy hitters. Banks, public entities, manufacturers — they all started entrusting their data flows to our furry friend. But the tale didn’t end there. In 2019, VMware reclaimed RabbitMQ by acquiring Pivotal, integrating it into the VMware Tanzu suite, thus bolstering its credibility even further by leveraging VMware’s deep pockets and expertise in not just crafting top-tier solutions, but also in their marketing finesse.
That’s when the magic happened. RabbitMQ evolved from a solid open-source project into an enterprise-grade messaging broker complete with robust features and round-the-clock support. It even bagged itself a snazzy new moniker: VMware Tanzu RabbitMQ.
Fast forward to 2023, and VMware has been brought under Broadcom’s wing, with RabbitMQ now a jewel in the crown of the Broadcom Tanzu portfolio. The air is thick with anticipation for the innovations this new chapter will bring, so stay tuned.
Not only a broker, but the best broker
I must confess: I have a fondness for RabbitMQ. Yet, my professional journey has also been rich with experiences with other messaging systems — special shout-outs to IBM MQ and Apache Kafka, both of which are outstanding in their own right! Now, let’s indulge in a thought experiment to illustrate my point. Imagine, if you will, a world without RabbitMQ (a grim thought, indeed).
Picture a company steeped in the use of general-purpose message-oriented middleware since the ’90s. The go-to choice? IBM MQ (once known as MQSeries), a staple for over three decades. Enter the era of IoT and M2M, bringing the need for MQTT. IBM, the original developer of MQTT, includes the protocol in IBM MQ. Yet, we’re aware of the existence of lighter, more popular MQTT brokers — making me question the economic sense of leveraging a heavyweight like an IBM MQ queue manager to serve as an MQTT broker. Sure, sticking with IBM MQ for MQTT is an option, or you could switch to Mosquitto, another commendable market player. Then there’s the demand to transmit vast amounts of data in our data-centric world. No fancy features such as routing are needed; just sheer volume. Kafka naturally springs to mind. What if legacy systems are in the mix? Think Mainframes or Tandem — for those who remember this one, a chuckle is warranted. Such systems might be inflexible regarding new dependencies or libraries; yet, they can communicate via telnet using the STOMP protocol. A bit of googling reveals several STOMP-supportive brokers like ActiveMQ, which also caters to JMS, HTTPS, and WebSockets. But the digital landscape is ever-evolving, and there’s a push towards open standards like AMQP. While IBM MQ and ActiveMQ support AMQP 1.0, the older AMQP 0.9.1 remains popular, even two decades on, likely due to its simplicity and broad community support. Options abound — Solace, SwiftMQ — each supporting both AMQP iterations. So, for these varied use cases, a company might juggle anywhere from three to six brokers just to shuffle data around.
But why not streamline? RabbitMQ is one broker to rule them all (please check my talk on this topic at CNCF Vienna 2023; while it may not be the presentation I’m most proud of, it was my first, and all beginnings are modest, so please be compassionate) . It supports AMQP 0.9.1 and 1.0, MQTT 3.1 and 5.0, JMS, WebSockets, HTTP/S, STOMP, and Streams — all within a single broker. This scenario is hypothetical, but the real-life narrative is even more compelling. RabbitMQ consistently outperforms in benchmarks and has seen remarkable improvements in recent years, especially since it has been under the wing of VMware (and now under Broadcom’s).
Yes, enterprise-ready
I’ve been working with very large enterprises for about 15 years now, and their demands are consistent: reliability, cost-effectiveness, and support. The latter sometimes translates to ‘I want someone to blame’ rather than ‘I seek a dependable partner for my critical systems.’
To clarify, Tanzu RabbitMQ is essentially the twin brother of open-source RabbitMQ; they share the same DNA, the same origin, and speak the same language, with subtle differences that give Tanzu RabbitMQ its unique edge (parents reading this might relate to having a favorite child, even if we know they’ll never admit it).
Reliability
This term can be interpreted in various ways. It might refer to product maturity, which is consistent across both versions of our furry broker. Or are we talking about service and data availability? Let’s be clear: losing data is a no-go. And if it happens, it’s only natural to seek accountability. However, with RabbitMQ, you likely won’t face this issue, and even less with Tanzu RabbitMQ. Thanks to its high availability features and built-in disaster recovery solutions, such as the Warm Stand-by replication and Schema Replication plugins (exclusive to Tanzu RabbitMQ), you can rest easy knowing your messaging system will function reliably, always. It’s all explained by the cool RabbitMQ team in here.
Pssst: you can replicate all data and schema from one primary cluster (upstream) to an infinite number of secondary ones (aka “DR clusters” or downstream)
Cost-Effectiveness
We haven’t delved deep into the high availability features I mentioned earlier, but we will later in this series. In short, what do you do when you want your service and data available at all times? You create copies. More instances mean more data copies, and the more copies you have, the safer you feel. However, transferring this data, particularly over a cloud network, incurs significant costs due to inter-zone traffic charges. And these can be hefty. Yet, the Tanzu team at VMware (now Broadcom) ingeniously included a Compression plugin for all inter-node communications, slashing traffic by up to 96%. For instance, a queue handling 2315 messages per second, this translates to savings of about $1000 monthly. Imagine the impact for a large financial institution processing over 200 billion messages daily — the savings could amount to over $1 million monthly, or roughly $13 million annually. Now that’s significant. Play a bit with the calculator and see for yourself!
Support
Oh, the support — no joking here, it’s a crucial aspect. Companies, even the large ones, are made up of people, and people are driven by emotions. Often, these emotions manifest as a desire to shift some of the responsibility elsewhere, and that’s completely natural. It’s a very human reaction. However, my advice to everyone is simple: make the most of the resources at your disposal. Support for any system should ideally mean having a reliable partner ready to assist in critical moments, allowing you to focus primarily on your business, not the intricacies of information systems. Beyond this, there’s not much else to say on support without veering into an overly promotional tone, and that’s not the goal here.
Wrapping Up
In my daily interactions with systems engineers, I’ve come to realize that many, even those seasoned in message-oriented middleware, aren’t fully aware of the enterprise-grade features available for RabbitMQ. The aim of this article wasn’t to push one product over another, as evidenced by my acknowledgment of other excellent products in the market. Rather, it’s about raising awareness. If you’re a systems engineer or a developer who’s stumbled upon this article, I hope it sparks your curiosity and encourages you to conduct your own research into this topic.
Following in this series
This article marks the beginning of the series. In the upcoming installments, we’ll get more technical. We’ll dive into the implementation of RabbitMQ/Tanzu RabbitMQ, explore best practices, tackle troubleshooting, and much more.
