Running Kafka Consumer using GoLang

GoLang is a widely used language past couple of years and many organisation are shifting to GoLang because of the speed, performance and the ease of development which can be seen in this article and short video attached with this.

This article focuses on how to setup a Kafka consumer in GoLang from scratch and covers from very basic to running Go application as Kafka Consumer. Once you have the Kafka brokers up and running the Go Program for Kafka consumer can be invoked to read the messages in real-time.

These simple 4 steps defines how to connect to Kafka Broker and read the messages.

1. The minimum Reader Configuration to Connect to Broker on a given topic.

2. Start a new Reader from the configuration

3. Start reading the messages in a loop.

4. Trigger the Kafka Consumer Go Program in a Go Routine to read the messages.

main.go file

package main

import (
   appKafka "./kafka"
   "fmt"
   "time"
)

func main()  {

   fmt.Println("Okay...")
   go appKafka.StartKafka()

   fmt.Println("Kafka has been started...")

   time.Sleep(10 * time.Minute)

}

kafka_config.go file

package kafka

import (
   "context"
   "fmt"
   "github.com/segmentio/kafka-go"
)
func StartKafka()  {
   conf := kafka.ReaderConfig{
      Brokers: []string {"localhost:9092"},
      Topic: "mytopic",
      GroupID: "g1",
      MaxBytes: 10,
   }

   reader := kafka.NewReader(conf)

   for {
      m, err := reader.ReadMessage(context.Background())
      if err != nil {
         fmt.Println("Some error occured", err)
         continue
      }
      fmt.Println("Message is : ", string(m.Value))
   }

}

The short video describes the whole process from scratch.