Circuit breaker pattern in Kafka consumer application

Introduction

In this article, we will see how to handle the failure scenarios in Kafka Consumer application. We will implement the Circuit breaker pattern in Kafka Consumer application using Apache camel and Spring boot.
It is required to know the basics of Kafka concepts, Apache camel and Springboot to proceed with this article.

Message processing challenge in Kafka
Imagine that we have a Kafka consumer application that consumes a message from a Kafka topic and send the data to an external service. If the service is down, the application should stop consuming any messages from Kafka topic. Also, the current batch of messages consumed from the topic should not be committed back to Kafka so that these messages can be processed again once the service is up again. We can use Circuit breaker pattern to handle this scenario.

What is Circuit breaker pattern ?
Circuit breaker is a design pattern implemented to check if the external service is available. It will block the application accessing the external service until it becomes available and implement any failure mechanism in the mean time. Once the service is back again, it will allow the application to access the external service as in normal behavior.

Apache camel
Apache camel is a powerful opensource integration framework to efficiently integrate with various systems to exchange messages. Camel has Kafka component to produce and consume to communicate with Kafka. It provides out of box features for Streaming, handling failure scenarios in Kafka etc.

Springboot
Springboot provides auto-configuration for Apache Camel. Auto-configuration of CamelContext auto-detects the Camel routes available in Spring context and registers the key Camel utilities. Spring boot property configuration also supports Camel properties.

ThrottlingRoutePolicy in Apache Camel
Configuring a route policy in a camel route is used to control the route during runtime. For example, you can define a route policy to stop the route during a failure condition. ThrottlingRoutePolicy is used to suspend or resume routes dynamically based on some conditions.

For our scenario, we want to stop consuming from Kafka topic when external service is down and resume processing once the service becomes available. We can configure the failure threshold and also the halfOpen time to check if service is available. We can call the healthCheck to see if service is available and resume message processing again as shown in the diagram here

Circuit breaker patter overview

Kafka Consumer implementation

Let us have two applications for this example:

1. Kafka Consumer
2. External Service

In Kafka Consumer application, we have Apache Camel and Springboot. It will consume message from a Kafka topic and send the message to the External service application which is implemented using Springboot.

Let us take a look at : https://gist.github.com/moulishanmuhavelu/92fbea2402d244f3617ed4466d4c99d6

The code snippet in the above link shows the Camel route implementation. We set a ThrottlingRoutePolicy implementation of RoutePolicy to the Camel route. We can also define the policy to be called only with the specific exceptions that we want. Only manual commit to kafka option is used so that we can control the message commit based on the availability of the external service.

A healthCheck call is implemented to check the availability of the external service when the circuit is open.

KafkaManualCommit implementation of Camel is used to do the manual commit to Kafka. It will check if this is the last record in the batch and then commit the batch offset to Kafka.

The external service is a simple Spring boot application exposing a REST endpoint. This application runs in the port that is configured. It exposes 2 endpoints, one for healthCheck and the other for process message. All the scenarios of the KafkaConsumer application can be tested by starting and stopping the external service application

Summary
In this article, we went through how to handle failure scenarios in Kafka consumer application to avoid any message loss.
We also saw how to make a Kafka consumer application to be resilient for the failure of downstream applications.

The source code of the consumer application is available in Kafka Consumer and the external service is available in Sample Service