Kafka Consumer in Python

Create a Python Kafka Consumer with pykafka and flask

(Source: own creation)

Since we learned how to produce data to Kafka, it is now time to write a Kafka Consumer with Python. Again we will use the pykafka client and again we start with a basic explanation of the pykafka consumer.

The first two lines should look familiar to you. After importing the KafkaClient (line 1) we are specifying the location of our Kafka Broker and assign it to the client variable (line 3).

Eventually, we can spin up our consumer with get_simple_consumer() which works only on a Kafka Topic. Furthermore, as a Kafka Topic usually contains a lot of data, we are looping through all messages in that topic with a loop (line 4). Finally, we are printing out the messages one by one (line 5). Important here → we must decode the Kafka messages as Kafka stores all messages in a Bytes format.

Now once we have a basic understanding of the pykafka consumer, let’s wrap an API around with Flask which we can call from the browser.

After importing Flask (line 1) we spin up a new Flask App (line 2) and start it at the end of the code block (lines 13+14). In…