BunnyBus : Building a Data Transit System

Introductions

I’ve had the pleasure of working at a few enterprise sized companies where data was treated as first class citizen. This is also true here at XO Group, where life of the data begins when the bride and groom register. This is followed by various steps of validation, transformation and data intelligence processing. Which eventually lands in the hands of our vendor(s) to capture leads or to help other couples find services based on similar matches.

I work in the Local’s team where I helped design a large portion of the data piping architecture. When I first joined the team, we used NServiceBus (NSB) in .NET to move messages around. Anyone having used NSB can appreciate the relative ease of creating an enterprise bus facade over MSMQ. As the years passed, our group needed to keep up with technology to make ourselves competitive both within XO and current technology culture. We finalized on Node.js and hapijs for all the benefits a single threaded event loop and a vibrant open source community would bring.

What is an enterprise data bus?

Quote from wikipedia.

An enterprise service bus (ESB) is implementing a communication system between mutually interacting software applications in a service-oriented architecture

The following are basic requirements for an ESB:

• Route message from publisher to any subscriber.
• Handle rejected messages.
• Persist data when subscribers have gone offline.
• Be fault tolerant.
• Provide for retry of data handling.
• Buffer I/O differentials between publisher and consumer.

Why is an enterprise data bus important?

We manage a microservice architecture which is a highly specialized form of SOA. Within this environment, we rely on queues to relay data from one application domain to another for requests that should be performed asynchronously. As a side note, we maintain homogenous data contracts for each queue. This can lead to an extraordinary amount of combinations for queues and routes needing to be created and maintained. In addition, these combinations are always in flux since services are brought in and out of play frequently.

For all the scenario above, overlaying an ESB facade just made sense.

Choosing RabbitMQ

As we first started porting our .Net stack over, XO Tech primarily used AWS SNS and SQS to bus data between teams (application domains). While this worked well for small amount of applications, the management of infrastructure did not scale well for Local’s application set where a single application domain had as many as 20 different events to publish with.

I started investigating different queue technology that would work with Node.js. I eventually landed upon RabbitMQ for the following reasons:

• Raw performance.
• FIFO (first in first out)
• Dynamic and flexible routing with route keys.
• Durable and fault tolerant.
• Built in realtime GUI for monitoring and debugging.

Using RabbitMQ

With any queue or data storage solutions, you have to work with low level drivers to get data in and out. Mostly, we aimed to find a driver that gave us the right abstractions away from the device and the right amount of feature. In, .Net, NSB afforded us the correct level of abstraction. In Node.js, using RabbitMQ equated to using the amqplib driver. This meant each of our applications would still need to manage connections, channels, exchanges and queues manually. This was when we decided to write our own Node.js module to gain the abstraction of NSB like capabilities.

BunnyBus

The original incarnation of BunnyBus came as flash of brilliance while discovering the power of Javascript. Hard work was placed on solidifying our microservice platform and the outcome is a module responsible for relaying all of our asynchronous traffic within Local’s ecosystem.

Much of our work would not have been possible without the huge amount of resources available in the open source community. In return, we want to share ‘BunnyBus`, our ESB facading module utilizing RabbitMQ.

Goals of BunnyBus

• Make it easy to bootstrap an enterprise bus to applications.
• Maintain high performance when compared to using amqplib directly.
• Facilitate flexibility in event subscription management.
• Create a developer-centric API.
• Support both Node.js callbacks and A+/Promise specifications.
• Automate exchange creation for publishing.
• Automate queue creation for both subscriptions and error handling for subscriptions.
• Automate connection and channel recovery when things become corrupt.
• Easily extend state management for the process manager.
• Allow extensions for listing event subscriptions.
• Create extensions for stopping subscriptions from being active with a simple call.
• Complete test coverage.
• Fully documented
• ES6 compliant.

How to get started

Simple, just npm i bunnybus

Definition of what a route is

A route is defined by a route key or event. A route key is something akin to a addressing system. A publisher has to provide one such as systemA.user-create. Any number of subscriber can opt to listen to the route key to get the message delivered to them. Subscribers can even ask BunnyBus to listen to all route keys like this systemA.* and all messages that are published with a matching value will be received.

What does a subscriber look like

So the code is pretty simple, configure a handler object that has a key which represents the name of event or route key. The value for this key is a function with the signature of message (content that was sent on the bus) and ack() (a function you use to clear the message off the bus). Lastly, using bunnyBus, subscribe() is what creates `queue1` and sets up the routing for messages to be delivered to this queue.

You can configure multiple route keys and handlers in the handler object so a queue handler for many different events. In addition, BunnyBus supports polymorphic payloads for types of String, Object and Buffer for the same event.

What does a publisher look like

Here, we created a payload with properties of event (containing route key) and body (just some data to send). Then we send it with publish(). Notice no queue is specified. But rest assured the message will be routed to the subscriber example above based on the event (route key) in the message.

There is More!

BunnyBus has a lot of power when wielded properly. Check out our documentation located within our Github repo.

How do we use it?

We combine the power of BunnyBus with flexibility of hapijs to build some awesome plugins that can be attached to any process allowing ESB capabilities to all of our applications. In additions, BunnyBus can support something like a notification system since subscriptions can be created with non durable and short lived queues so messages can be routed to Socket.io edged services.

About the Author

Lam Chan

Lam is a Software Architect for the Locals Squads @ XO Group. He is a seasoned polyglot engineer with over 16 years of professional
experience working with startups and multiple fortune 500 companies. When he is away from the office, he enjoys contributing to OSS
projects and dabbles with wood working projects. Find out more about Lam on LinkedIn