Introduction to Kafka with Nodejs

Hi friends, In this blog, I will give glimpses of Kafka with Nodejs.

https://www.photopea.com

What is Kafka

Kafka is the data pipeline that reliably processes and moves data from one system to another. It uses the Pub/Sub method for data communication. Once the Producer sends data to the Kafka broker, it keeps the data until the consumers receive it.

Instead of Pub/Sub, we can call it Producer and Consumer. When we are developing microservices we can use Kafka for communication between services.

In the below example, we can see the Kafkajs with node application.

Kafka Example:

Create a project setup and installation

mkdir kafka-tutorial
cd kafka-tutorial
$ npm init -y
$ npm install kafkajs --save

Create a Producer

// producer.js
const { Kafka, Partitioners } = require('kafkajs');

const kafka = new Kafka({
  brokers: ['localhost:9092']
});

const producer = kafka.producer({
  createPartitioner: Partitioners.LegacyPartitioner
});

const msg = {
  name: 'Test',
  timestamp: 1660199167
};

async function start() {
  await producer.connect();
  await producer.send({
    topic: 'hello',
    messages: [
      {
        key: 'key1',
        value: JSON.stringify(msg)
      }
    ]
  });
}

start();

A producer can send data to multiple brokers. After establishing a connection with the Kafka broker we created a producer. Once the producer is connected then the producer can send data to the broker. We have to set up a topic the consumer can receive using the topic.

Create a Consumer

// consumer.js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  brokers: ['localhost:9092']
});
const consumer = kafka.consumer({ groupId: 'test-1' });

async function start() {
  await consumer.connect();
  await consumer.subscribe({ topic: 'hello' });

  await consumer.run({
    eachMessage: async ({ topic, partition, message, heartbeat }) => {
      console.log({
        value: message.value.toString()
      });
      await heartbeat();
    }
  });
}

start();

A consumer can receive data from producers. When we create a consumer we can configure groupId. If multiple consumers are connected with the same groupId then any of the consumers only receive data from the producer. Using this feature we can run the consumers in cluster mode. Based on the topic consumer receives value from producers.

Run the application

Open the terminal in the one tab and run consumer.js

$ node consumer.js

Open another tab of the terminal and run producer.js

$ node producer.js

Sample Output:

Sample output

Thank you for reading. Have a nice day!