Say Hi to Delivery Boy!
A simple, convention-based library for publishing messages to Kafka from Rails.
This time, it’s all about message delivery.
Up until now, if you wanted to produce messages to Kafka from a Rails application, you’d need to write some boilerplate code: instantiating the ruby-kafka client with your desired configuration, setting up a producer instance with the right message delivery policy, and making sure to shut things down properly upon exit, including making sure to deliver any pending messages.
It was never a lot of work, but it was still an impediment to getting started, and there were too many chances to mess things up. That’s why we’re releasing Delivery Boy, a simple library that helps you publish messages to Kafka. It reads configuration from a YAML file (or environment variables, if you’re into that) and handles setting up thread-safe Kafka producer instances for you.
Add this to your Gemfile:
bundle install followed by
bundle exec rails generate delivery_boy:install. This will generate a config file at
config/delivery_boy.yml. If your Kafka broker is running locally, this should work out of the box. Otherwise change the file to point at one or more of your brokers.
Once this is done, you can start publishing messages to Kafka immediately! Here’s an example:
This will push the pending message to a background thread that handles message deliveries automatically. Doing this is important if you want to insulate your application against Kafka broker failures: while it is possible to deliver messages to Kafka directly using the
deliver method, doing so would mean that a delivery error would blow up your whole application. We recommend using
deliver_async unless you’re absolutely sure that it’s okay for your Rails code to get an exception — or maybe hang for a while — when Kafka is having issues. Better safe than sorry.
At Zendesk, we’re already using DeliveryBoy at scale, producing between 1,000 and 2,000 messages every second in our global deployment.
If you want to read more about DeliveryBoy, head over to its GitHub page!