Introducing Racecar

A simple framework for writing Kafka consumers in Ruby

At Zendesk we’ve invested heavily in Kafka. We’ve authored the Maxwell MySQL change capture application and the ruby-kafka client library.

Today we’d like to announce a new open source project: Racecar. Racecar makes it dead easy to write, configure, test, and run Kafka consumers in Ruby, and integrates nicely into Rails applications. It sacrifices some flexibility in exchange for simpler operation — if you need more tuning, you can always use ruby-kafka directly.

If you’re using Rails, getting started is easy; just add this to your Gemfile:

gem "racecar"

Now, add a consumer class that will receive messages from your Kafka topic. Start by executing the following:

$ rails generate racecar:consumer hello

This should create a file at app/consumers/hello_consumer.rb that looks like this:

Let’s test things out! Assuming you’re running Kafka on your local machine, execute the following command:

$ racecar HelloConsumer

The consumer will start running, waiting for messages to appear on the topic some-topic. Let’s write a message to this topic in order to test that things actually work. In another terminal window, execute the following:

$ racecarctl produce --value "hello!" --topic some-topic

You should now see the greeting printed by your consumer!

If you’re not running a Kafka broker on your local machine, either execute export before launching the consumer with Racecar or read the documentation on how to configure Racecar for production and non-default environments.

You can start multiple instances of HelloConsumer — they will form what’s known as a consumer group and coordinate processing of messages, making sure that each partition in a topic is handled by just a single consumer.

Note that Rails is not required to use Racecar. Check out the documentation for more information.

Learn more about Racecar on GitHub and tell us what you think!

Racecar goes wroom!!