Apache Camel: A route to integration

Anurag Lanjewar
Globant
Published in
4 min readNov 6, 2023
Photo by Bernice Tong on Unsplash
Photo by Bernice Tong on Unsplash

We now live in a world of distributed and integrated Applications. With that comes the great responsibility of communicating between these applications without losing data integrity and fault tolerance.

Apache Camel is an open-source project widely used to implement the Enterprise Integration Pattern. We can use both Java code and Spring XML configurations to enable different integrations.

In this blog post, we will take a look at why to use Apache Camel and a simple example of Apache Camel connecting to Kafka as a producer and a consumer.

Why Apache Camel?

If you want to integrate with multiple systems and do not want to invest much time in handling configurations to get the data and process the data, Apache Camel is what you need. It allows end users to integrate various systems using the same API. It has support for multiple protocols and data types. Camel uses URI for endpoint resolution, so it's very easy to work with any kind of transport, such as HTTP, REST, JMS, web service, File, FTP, TCP, Mail, and many others.

Connecting Kafka via Apache Camel

We will be taking the example of connecting to Kafka as a Producer and consuming the data as a Consumer.

For a Camel Kafka example, we will require the following dependencies

<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
</dependency>

<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-kafka</artifactId>
</dependency>

For the Spring boot project, you can add a Camel-Kafka starter

<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-kafka-starter</artifactId>
</dependency>

Route Builder

The main component to build the routes is the RouteBuilder. A RouteBuilder is used to define the whole workflow, like how to receive the data and what are the processing pipelines and we can even decide if we want to process these data differently based on parameters present in the data.

For this example, we can use both the normal and Spring way. Just adding @Component to your route builder class will add the route in the Camel context on application startup.

@Component
public class CustomRouteBuilder extends RouteBuilder {

@Override
public void configure() throws Exception {

// Kafka Producer using the file data
from("file://C:/anyDirectory?fileName=MyFile.txt&noop=true")
.doTry()
.process(MyProcessor.class, "handlerMethodName"))
.to("kafka:myTopic?brokers=localhost:9092");
.doCatch(IOException.class, IllegalStateException.class)
.to("mock:catch")
.doFinally()
.to("mock:finally")

// Kafka Consumer
from("kafka:myTopic?brokers=localhost:9092")
.log("Data received from Kafka : ${body}");

// Camel's error handler can handle the exception and send it to a different queue
onException(Exception.class).process(ExceptionLogger.class, "handlingMethodName")
.handled(true)
.to("rabbitmq:failureQueue");
}
}

Datasource

Various data sources can be used in the above example. It can be a file, rest endpoint, JMS, can be another route, and many more. You can check here the different components provided by Apache Camel and choose as per your need.

onException

onException is used to define the exception that can occur and how to handle it in .process we can pass the class name and the handling method name separated by a comma. By setting .handled true, we are marking that the original exception has been cleared and whatever response from the handling method is going to be returned. The .to is used to send the handled response to the final pipeline or subsequent route. doTry and doCatch can be used to handle the exception at route level

from

from will be the starting point of any Camel route. In this example, we have taken file input from local storage. You can use many Apache Camel pre-defined components as per your use case. You can refer to the Camel core components lists here.

Code Flow

The producer will listen to the file data, and you can marshal and unmarshal the data if needed; the processor class’s handler method can transform the data as per your need and then send it to Kafka.

The consumer route is parallelly listening to the topic and logs it to the console.

Camel Route Context

The route gets created in the Camel Context; these are created at the startup and can be used for the whole Application lifecycle. We can add new Routes at the runtime and refresh the context to reflect the new route changes. Similarly, enabling or disabling routes in a running environment is possible!

Circuit Breaker

Below is a sample route showing how easy it is to create a Fault Tolerance EIP circuit breaker with fallback.

from("direct:start")
.circuitBreaker()
.to("http://myservice.com")
.onFallback()
.transform().constant("Call to service failed!")
.end()
.to("mock:end");

source: https://camel.apache.org/components/3.21.x/eips/resilience4j-eip.html

Configurations

Apache Camel gives us various configurations like Throttling, Back pressure, Aggregation strategies, etc. So if our consumers are slow, we can use Back pressure configuration to limit the incoming messages from consumers.

Aggregation Strategy

We can aggregate the results to use them with batch processing or send the data individually. This gives us the flexibility to integrate with the consumers based on their processing capabilities. In case we want to persist our data in a high-throughput database, it will be very good to aggregate the records and persist, but on the other hand, if we have a slow database, we can aggregate to a smaller batch size. For further information, you can refer to this document by Apache Camel.

Queues

We can even use the decentralised way of processing by introducing a Queue (e.g.: RabbitMQ or ActiveMQ) or can use an in-memory SEDA queue. It's a matter of choosing the right tool as per your needs.

Conclusion

Apache Camel comes with various options to help us integrate with different systems. Various configurations are available with Apache Camel, which makes it highly customizable; the libraries provided for these integrations are readily available as we have used the Camel-Kafka in our example. There is a small learning curve to understand all the features available to us to utilise them efficiently.

Useful links

You can refer to the below document for different configurations available for Camel Routes:- https://camel.apache.org/manual/route-configuration.html

For throttling-related configurations, refer to the below document:- https://camel.apache.org/components/4.0.x/eips/throttle-eip.html

You can refer to the below example for Rest integration with Apache Camel:- https://www.javadevjournal.com/spring-boot/apache-camel-spring-boot/

--

--