Getting Started with Kafka in Spring Boot: Producing and Consuming Messages

Kafka is a powerful distributed streaming platform that allows you to build real-time applications. In this article, we’ll explore how to use Kafka with Spring Boot for producing and consuming messages. We’ll cover the basics and provide code examples to help you get started.

Prerequisites

Before we begin, make sure you have the following installed:

• Java JDK
• Maven
• Kafka

Setting Up a Spring Boot Project

Let’s start by creating a new Spring Boot project using Spring Initializer. Include the Spring Web, Spring for Apache Kafka, and Lombok dependencies.

Producing Messages

Step 1: Configure Kafka Properties

Open application.properties and configure Kafka properties:

spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

Step 2: Create a Kafka Producer

Create a Kafka producer class:

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class KafkaProducerService {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void sendMessage(String topic, String message) {
        kafkaTemplate.send(topic, message);
    }
}

Step 3: Sending Messages

Now, you can send a message using the KafkaProducerService:

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/kafka")
public class KafkaController {

    @Autowired
    private KafkaProducerService producerService;

    @PostMapping("/send/{topic}")
    public void sendMessage(@PathVariable String topic, @RequestBody String message) {
        producerService.sendMessage(topic, message);
    }
}

Consuming Messages

Step 1: Create a Kafka Consumer

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Service
public class KafkaConsumerService {

    @KafkaListener(topics = "my-topic", groupId = "my-group")
    public void consume(String message) {
        System.out.println("Received message: " + message);
    }
}

Step 2: Configure Consumer Properties

Open application.properties and configure consumer properties:

spring.kafka.consumer.bootstrap-servers=localhost:9092
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.group-id=my-group

Step 3: Start Consuming

Start your Spring Boot application, and it will listen for messages on the specified topic.

Produced Message

2023-10-13 14:30:00.123  INFO 12345 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 2.8.0
2023-10-13 14:30:00.123  INFO 12345 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 1234abcd
2023-10-13 14:30:01.234  INFO 12345 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'taskScheduler'
2023-10-13 14:30:02.345  INFO 12345 --- [ad | producer-1] org.apache.kafka.clients.Metadata       : [Producer clientId=producer-1] Cluster ID: GsyfGkljQ1Kdb1iK4h4dDA
2023-10-13 14:30:03.456  INFO 12345 --- [ad | producer-1] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 2.8.0
2023-10-13 14:30:03.456  INFO 12345 --- [ad | producer-1] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 1234abcd
2023-10-13 14:30:04.567  INFO 12345 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Shutting down ExecutorService 'taskScheduler'

Consumed Message

Received message: Hello, Kafka!

Conclusion

Congratulations! You’ve learned how to produce and consume messages using Kafka in a Spring Boot application. This is just the beginning; you can now build more complex real-time applications using Kafka’s powerful features.

For more detailed documentation and advanced configurations, refer to the official Spring Kafka documentation.

Happy coding!