Monitoring and Reporting the real-time data in Power BI

3 min readMay 14, 2020

About Kafka:
Apache Kafka is an open-source stream-processing software platform developed by LinkedIn and donated to the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka can connect to external systems (for data import/export) via Kafka Connect and provides Kafka Streams, a Java stream processing library.

Kafka has 3 major components namely,
1) Topics
2) Producer
3) Consumer
In real-world we can explain Kafka in simple terms with YouTube
Let the YouTube channel be the topic and producer is the creator of the content in the channel and subscribers are the targeted audiences for the channel. Topic is the passage or channel through which messages are communicated.

Kafka is used for these broad classes of applications:

Building real-time streaming data pipelines that reliably get data between systems or applications.

Building real-time streaming applications that transform or react to the streams of data.

Let me explain the project,

Kafka Configuration:

1. First, we need to install Kafka platform in our Cloudera/hortonworks instance https://kafka.apache.org/downloads

2. Then in the downloaded folder, go to the config option and modify the server.properties files for the 2 brokers of 2 id’s and 2 different port numbers (This is because as we are trying to establish a connection of producer and consumer in the same machine we are following different port numbers)

3. It should be accessed by vi config/server.properties

4. Then we need to start the zookeeper

5. Then we must start both the servers that you have configured. (server1.properties and server2.properties)

6. Create a topic to transfer messages.

7. Once a topic is created, configure producer and consumer

8. In the producer we have mentioned the broker list whereas in consumers we need not mention. This is because we had configured the zookeeper which maintains all the offsets of the messages.

9. Now the messages can be transferred easily.

https://youtu.be/DMzEq_XhZkY

Configuration in PowerBI:

1. In PowerBI workspace -> go-to dataset -> Create Streaming dataset -> api -> fill the details -> copy that API link to configure in python code.

2. Please find the below python codes for the transfer of messages to PowerBI in real-time.

Python Producer