Live Streaming Data Using Kafka, Node JS, WebSocket and Chart JS

Yeah well the title of the story pretty much define what we were trying to achieve. This was a small POC that we started as a stepping stone to handle a huge data coming from our client and provide a live streaming after performing some processing.

I wanted to share my experience and give a walk through to anyone who would want to create a simple Live streaming of data using Kafka, Node JS, WebSocket and Chart JS :)

The reason for writing down this story is not to define something which might sound quite obvious but to allow reader to be able to follow the steps and have a working example. I found articles related to this topic on google but hardly found a step to step instruction that i could follow. And being new to Kafka, Node JS it could be daunting.

Follow the guide below and you should be able to have something like this up and running in minutes.

  1. Start with the basic Kafka installation on your machine using

Use the steps mentioned here to set-up and start Kafka server, create a topic and then use the Kafka console producer to send messages to the topic.

That mean do as said from Step 1 to Step 3 as in the Kafka quick-start document. Further we will use Step 4 to send some messages.

I have created a topic say — test1 and have my producer ready as below

2. For creating a consumer using Node JS, we will be using . The purpose is straightforward — read messages from the Kafka topic. Following is a small snippet of code to create a consumer for the topic test1 that we created earlier in step 1. The topic I created is available on localhost:9092, modify your code according to your settings. (Just keep reference of this code in mind, we will be using it further)

3. On the Node JS service side we will be creating an WebSocket API using, which will be returning back data to the client (browser). A simple example would be as following:

Step 3 Continued :

The purpose is to combine the Kafka consumer and the WebSocket API to play together and return data coming from Kafka producer. So combining the above two steps will result in something like this, let’s call it consumer.js.

Use command- node consumer.js to keep everything up and running

4. At the web browser (our client) we would invoke the WebSocket API and display the data as using Chart JS.

Whip out a index.html that would just include a placeholder for the chart and some basic script references:

In the index.js we would create a basic line chart and invoke the WebSocket API as shown below:

Open the index.html on any browser, We are all ready to check the live streaming of data by sending messages using Kafka console producer. The data would be shown as a line graph which will keep on updating as the messages are send.

See no rocket science involved.

This is the most basic example and in the upcoming stories I will be doing refactoring and make a more robust system. Hope this current draft allow you to create a simple example and understand end to end flow.