A simple framework for writing Kafka consumers in Ruby
Today we’d like to announce a new open source project: Racecar. Racecar makes it dead easy to write, configure, test, and run Kafka consumers in Ruby, and integrates nicely into Rails applications. It sacrifices some flexibility in exchange for simpler operation — if you need more tuning, you can always use ruby-kafka directly.
If you’re using Rails, getting started is easy; just add this to your Gemfile:
Now, add a consumer class that will receive messages from your Kafka topic. Start by executing the following:
rails generate racecar:consumer hello
This should create a file at
app/consumers/hello_consumer.rb that looks like this:
Let’s test things out! Assuming you’re running Kafka on your local machine, execute the following command:
$ racecar HelloConsumer
The consumer will start running, waiting for messages to appear on the topic
some-topic. Let’s write a message to this topic in order to test that things actually work. In another terminal window, execute the following:
$ racecarctl produce --value "hello!" --topic some-topic
You should now see the greeting printed by your consumer!
If you’re not running a Kafka broker on your local machine, either execute
export RACECAR_BROKERS=my.kafka.host.name:9092 before launching the consumer with Racecar or read the documentation on how to configure Racecar for production and non-default environments.
You can start multiple instances of
HelloConsumer — they will form what’s known as a consumer group and coordinate processing of messages, making sure that each partition in a topic is handled by just a single consumer.
Note that Rails is not required to use Racecar. Check out the documentation for more information.
Learn more about Racecar on GitHub and tell us what you think!