Monitoring KafkaJS with Prometheus in a Node.js application

Photo by iSawRed on Unsplash

In my previous story we were looking into why monitoring is important to build a great product and how you can monitor the Node.js MongoDB Driver. In this article we will look into how you can monitor your Kafka application in a Node.js application using KafkaJS and Prometheus.

Step 1: Add dependencies
Add the kafkajs-prometheus-exporter dependency to your project which enables monitoring of KafkaJS.

npm i @christiangalsterer/kafkajs-prometheus-exporter

If not done yet, please add the dependency for prom-client too.

npm i prom-client

Step 2: Set up the kafkajs-prometheus-exporter
After adding the dependencies set up your kafkajs client and register the exporter with the prom-client.

With Typescript you can start using the following example:

import { Kafka } from "kafkjs";
import { Registry, collectDefaultMetrics } from "prom-client";
import { monitorKafkaJSProducer, monitorKafkaJSConsumer, monitorKafkaJSAdmin } from "@christiangalsterer/kafkajs-prometheus-exporter";

...

// set up the KafkaJS client
const clientId = 'myClientId'
const kafka = new Kafka({
    clientId: clientId,
    brokers: ['localhost:9094'],
})

const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'myGroupId' })
const admin = kafka.admin()

// set up the prometheus client
const register = new Registry();
collectDefaultMetrics({ register })

// monitor KafkaJS producer
monitorKafkaJSProducer(producer, register, { defaultLabels: {client_id: clientId} })

// monitor KafkaJS consumer
monitorKafkaJSConsumer(consumer, register, { defaultLabels: {client_id: clientId} })

// monitor KafkaJS admin
kafkaExporter.monitorKafkaJSAdmin(admin, register, { defaultLabels: {client_id: clientId} })

...

// connect to Kafka *after* calling monitorKafkaJSProducer() / monitorKafkaJSConsumer / monitorKafkaJSAdmin
await producer.connect()
await consumer.connect()
await admin.connect()

With Javascript you can start using the following example:

const Kafka = require('kafkajs')
const promClient = require( 'prom-client');
const kafkaExporter = require('@christiangalsterer/kafkajs-prometheus-exporter')

...

// set up the KafkaJS client
const clientId = 'myClientId'
const kafka = new Kafka({
    clientId: clientId,
    brokers: ['localhost:9094'],
})

const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'myGroupId' })
const admin = kafka.admin()

// set up the prometheus client
const collectDefaultMetrics = promClient.collectDefaultMetrics;
const Registry = promClient.Registry;
const register = new Registry();
collectDefaultMetrics({ register });

// monitor KafkaJS producer
kafkaExporter.monitorKafkaJSProducer(producer, register, { defaultLabels: {client_id: clientId} })

// monitor KafkaJS consumer
kafkaExporter.monitorKafkaJSConsumer(consumer, register, { defaultLabels: {client_id: clientId} })

// monitor KafkaJS admin
kafkaExporter.monitorKafkaJSAdmin(admin, register, { defaultLabels: {client_id: clientId} })

...

// connect to Kafka *after* calling monitorKafkaJSProducer() / monitorKafkaJSConsumer / monitorKafkaJSAdmin
await producer.connect()
await consumer.connect()
await admin.connect()

Step 3: Start monitoring

Congratulations!!! You have completed all preparation steps and can start monitoring KafkaJS in your Node.js application.

An example dashboard for Grafana is available here displaying the provided metrics by the exporter.

Here an example for producer metrics:

Here an example for consumer metrics:

Here an example for admin metrics:

Available Metrics

The complete list of available metrics can be found here.

Configuration

The exporter can be configured in a couple of ways. The complete list of configuration options can be found here.