Performance of Kafka Consumers: 1 Billion messages

Trendyol has many services and most of them communicate with each other asynchronously. The critical point is Kafka. It's located at the center between services. Some topics have 100–200 events some topics have 100 million-200 million events. As an inventory team, we inform stock and price via Kafka events to another team with huge numbers. The story has begun here.

I was working with Python consumers on my previous team. Now, I work with Go consumers on my new team. End of the day, we have two programming languages and platforms as options for using consumers. Which one is the best suitable for our domain or task? This is the best option to learn which one is best: Benchmark. Let's find out the answer together.

My current domain team has high traffic. That's why I will benchmark with high traffic in this article and we will figure out whether Python or Go is best.

How do we do?

Libraries I used for these projects:

• kafka-python
• kafka-go

Now, let's create a consumer in Python and Go. Consumers will search in Topic with the specific key I provided. Consumers will subscribe to the topic. I will create threads equal to the partition count of the Topic. They will run as parallelly.
For instance, a topic has ten partitions. Then I will create ten threads. Each one assigns to a Partition and consumes events until the last event.

Let's review the codes.

Python:

class ConsumeHandler():
    def __init__(self, kafka :KafkaService, redis: RedisService):
        self.kafka = kafka
        self.redis = redis
    
    def create_consumers(self, search_data : SearchData):
        topic = search_data.topic_name
        consumer = self.kafka.get_consumer()
        partitions = consumer.partitions_for_topic(topic)
        consumer.close(False)
        
        started_time = datetime.utcnow().timestamp()
        self.redis.started_search(search_data.id)
        partitions_list = []
        if partitions:        
            for partition in partitions:
                partitions_list.append(TopicPartition(topic, partition))

            random_id =str(uuid.uuid4())    
            data = []
            for partition in partitions_list:
                my_dict = {"kafka_name" : self.kafka.get_name(), 'p' : partition, 's_d' : search_data, 'r_i' : random_id}
                data.append(my_dict)
            
            with Pool(len(data)) as p:
                p.map(assign_consumer, data)
                
        started_time = datetime.utcnow().timestamp() - started_time
        self.redis.time(search_data.id, started_time)
        
            
def assign_consumer(dict : Dict):
    kafka = get_kafka_factory(dict["kafka_name"])
    consumer = kafka.create_consumer(dict['r_i'])
    consumer.assign([dict['p']])
    listener = Listener(consumer, dict['p'], dict['s_d'])
    listener.consume(datetime.utcnow())  

Go:

func (s *consumeHandler) handle(searchData *model.SearchData) {
 pIds := s.getPartitionIds(searchData)
 if pIds == nil || len(pIds) == 0 {
  return
 }

 var wg sync.WaitGroup
 wg.Add(len(pIds))
 sd := time.Now().UnixMilli()
 s.redis.SearchStarted(searchData.Id)
 for i := range pIds {
  pId := pIds[i]
  go func(sd *model.SearchData, p int, r *client.RedisService) {
   h := NewListenerHandler(sd, s.c, p, r)
   h.Handle()
   wg.Done()
  }(searchData, pId, s.redis)
 }

 wg.Wait()
 s.redis.SearchFinished(searchData.Id)
 sd = time.Now().UnixMilli() - sd
 s.redis.Time(searchData.Id, sd)
}

We are ready to fight!

As our projects are ready. We can start the benchmark. Our test will execute on a Kubernetes pod that has a 5000m CPU and 2 GB ram.

During the benchmark, we will check three parameters and we will judge them.

• Memory usage
• CPU usage
• Execution Time

First Round: First hit

Topic Size: 11.8 GB
Message Count: 18.512.979
Partition Count: 10
Founded search key in the topic: 105

Grafana metric of Python:

Grafana metric of Go:

The results of the benchmark belong to the first round. The winner is Go. I can't find words to say. They’re chalk and cheese. There are differences in the usage of resources, but also in execution time as %300 nearly.

The current mood of Python:

Second round: More events

Topic Size: 110 GB
Message Count: 172.545.102
Partition Count: 35
Founded search key in the topic: 876.142

Grafana metric of Python:

Grafana metric of Go:

When numbers increased, CPU usage is equal nearly. But about memory, Python is knocked out. The execution time is the same as the previous one.

The current mood of Go:

Third round: Last dance

Topic Size: 2.15 TB
Message Count: 1.384.763.382
Partition Count: 30
Founded search key in the topic: 2

Grafana metric of Go:

This benchmark happens alone because python is knocked out. I can't wait for Python to finish the test. It may take a long time for these resources.

Again the current mood of Go:

Maybe Go’s performance overshadows Python, but Python is developer friendly. Of course, I can choose Python in these scenarios. If I don't need a performance or need to develop minimized consumer or I need to one-time run/remove console app. But when I need performance or huge numbers, I would rather Go without question.

Next time I will try to find a new fighter who can handle it until the last round against Go.

Thank you for reading until here. See you next in my article.

THE END