Dead Letter Queues (DLQs) in Kafka

Photo by Safar Safarov on Unsplash

What is Dead Letter Queue ?

A dead letter queue is a simple topic in the Kafka cluster which acts as the destination for messages that were not able to make it to their desired destination due to some error.

What is the purpose of using DLQs ?

Let’s say we have a kafka consumer-producer chain that reads messages in JSON format from “source-topic” and produces transformed JSON messages to “target-topic”.

The below scenarios explain the need for DLQs:

• A message on “source-topic” was not a valid JSON format so could not be deserialized by the consumer. As the message is not in valid format it cannot be transformed and published to “target-topic”. It’s better to log such malformed messages to a “dlq” target topic from where the malformed messages can be analysed later without interrupting the flow of other valid messages.
• An error occurs while processing a message from the “source-topic”. This might occur when the message is in a valid JSON format but the data is not as expected. The simplest example is if the message has a field for age which is expected to be positive, but we have received “age”: “-30” in the message. Such messages should be logged to “dlq” topic for further analysis.
• “target-topic” is full so cannot accept any new messages. This might happen if the load on your topic is very high. In this case we can have a target “dlq” topic for such messages.
• “target-topic” does not exists. Thus we don’t have any destination for the messages and a possibility of message loss. While this scenario rarely occurs, it’s better to have some target topic for such messages.

How to implement the concept of DLQs in Kafka ?

• If you are using Kafka Connect then this can be easily setup using the below configuration parameters. For more details visit the confluent blog.

errors.tolerance = all
errors.deadletterqueue.topic.name = <topic name>

• If you are using Kafka Streams then try setting the below property given in the documentation.

default.deserialization.exception.handler

• If you have your own producer and consumers then surround your kafka consumer logic inside try-block and if any exception occurs send the message to “dlq” topic. If there is no error send the message to “target-topic” after transformation. Below is the sample code for this scenario implemented in Python:

consumer_with_dlq_logic.py

So for your system to be reliable and resilient you should have DLQs and there are multiple approaches of implementing DLQs in Kafka.