Welcome to Kafkaland!

Intro

If you ever worked with message queueing or streaming platforms, Kafka is one that you should check. It’s best use is for real-time data streaming.

In case you’re considering Kafka for your application, Kafka is best for the following use cases:

• Pub/Sub messaging
• Stream processing
• Data storage

First of all, before touching Kafka, one should know the following: Kafka is blazing fast, you have enormous freedom, and it’s easy to go wrong with it when coming from another message queuing platform.

In this article we will focus on doing the things right with Kafka.

Key Concepts:

• message: unit of data
• topic: data channel
• partition: topics can have more partitions
• offset: position of a message in a partition
• producer: allows an application to publish data to a given topic
• consumer: the application can consume data from one or more topics. One topic can have multiple consumers. Consumers can subscribe to any number of partitions of a topic.
• consumer groups: consumers can be organized in consumer groups, that are reading from a topic, in parallel when reading from different partitions
• broker: manages the storage of topics
• cluster: multiple brokers, containing several brokers
• leader/replica: each broker contains a number of partitions of a topic, each of these can be a leader or a replica. All writes and reads to a topic are performed on the leader, which coordinates updating the replicas. If a leader fails, a replica takes the role of the leader

More on Kafka basics can be found here.

Basics with Node.js

For this intro we’ll use Node.js, and we’ll create basic producer and consumer.

First of all, you’ll be installing Kafka locally as described here.

This article assumes that we have only one broker.

Topic(s) will be created before our application is run, see below.

At the moment of writing this intro, there are 3 stable packages to use with NodeJS:

1. Node-rdkafka
2. KafkaJS
3. Kafka-node

All of these work in a similar way, for introduction we’ll use node-rdkafka (npm install node-rdkafka)

Assuming, you’ve installed Kafka locally with default settings and started one broker, we can create a simple producer and consumer.

Create the topic manually:

Create producer, and produce an event:

Create the consumer and consume event(s):

This is an example for a flowing consumer. It consumes events as they appear on the topic. Another possibility is to use a non-flowing consumer, by specifying the number of events to consume as parameter to consumer.consume(number). In this case, we’ll get the messages after having number unconsumed messages in the topic.

Commits

Since Kafka was designed for fail safety, consumers can mark the messages as consumed — committed. By default, node-rdkafka does auto-commits, also, it can be done manually: consumer.commit() or consumer.commitMessage(message).

In case of the consumer disconnected, another consumer can continue from the committed message.

Welcome to Kafkaland! Enjoy!

This is the first from a series of four articles. Will follow up with specific cases of Kafka usage for: microservices, chatbots and fintech projects. Yay!