How to use Apache Kafka without Java client?

Introduction

Apache Kafka was created with a client API only for Scala and Java. After that initial period, the Kafka Client API has been developed for many other programming languages.

So, what if you want to use a development language that is unavailable? Or what if you want to access from an api gateway to Kafka?

In TUI Musement, we experienced the following scenarios:

• Use Kafka from SAPFI (SAP Financial Accounting, developed in ABAP)
• Use Kafka from Apigee (the api gateway we are using)

The solution we applied was to use Confluent Kafka Rest Proxy.

What is Confluent Kafka Rest Proxy?

Confluent Kafka Rest Proxy is an open source package developed by Confluent that allows users to access Kafka trough a RESTful API.

The API supports many interactions with your cluster, including message production and consumption, access to cluster metadata as well as management operations.

We can use it with binary, json or avro message. It also integrates well with the Confluent Schema Registry.

It is available in various format with easy installation.

How does the Confluent Kafka Rest Proxy work?

HTTP wrapper of Java libraries

It simply wraps the existing libraries provided with the Apache Kafka open source project. This includes not only the producer and consumer you would expect, but also access to cluster metadata and admin operations. Therefore, it depends on specific versions of the Kafka libraries and the Confluent Platform packaging ensures the REST Proxy uses a compatible version.

JSON with flexible embedded data

Their requests need to include embedded data, the serialized key and value data that Kafka deals with. To make this flexible, confluent use vendor specific content types in Content-Type and Accept headers to make the format of the data explicit.

Stateful consumers

Consumers are stateful and tied to a particular proxy instance.

Distributed and load balanced

Despite consumers being tied to the proxy instance that created them, the entire REST Proxy API is designed to be accessed via any load balancing mechanism (e.g. discovery services, round robin DNS, or software load balancing proxy).

Examples

Publish a message

To publish a message you just need to send a json message with the following structure.

NOTE: you can publish one or various records at the same request (note that records field is a list).

Request

curl --location --request POST 'https://kafka-rest-cmn.test.tui-dx.com/topics/suppliers' \
--header 'Content-Type: application/vnd.kafka.json.v2+json' \
--data-raw '{
  "records": [
    {
      "key": "1",
      "value": {
        "id": "1",
        "customer_code": "1",
        "telephone": "888582158",
        "email": "supplier1@test.com",
        "language": "EN"        
      }
    }
  ]
}'

Response

{
  "offsets": [
    {
      "partition": 0,
      "offset": 3,
      "error_code": null,
      "error": null
    }
  ],
  "key_schema_id": null,
  "value_schema_id": null
}

Consume published messages

Create a consumer

You need to create a consumer in a consumer group

Request

curl --location --request POST 'https://kafka-rest-cmn.test.tui-dx.com/consumers/sapfi' \
--header 'Content-Type:  application/vnd.kafka.v2+json' \
--data-raw '{
"name": "sapfi_consumer_1",
"format": "json",
"auto.offset.reset": "earliest",
"auto.commit.enable": "true"
}'

Response

{
"instance_id": "sapfi_consumer_1",
"base_uri": "http://kafka-rest-cmn.test.tui-dx.com/consumers/sapfi/instances/sapfi_consumer_1"
}

Create a subscription for a consumer

You need to create a subscription indicating the list of topics where you want to consume messages.

Request

curl --location --request POST 'https://kafka-rest-cmn.test.tui-dx.com/consumers/sapfi/instances/sapfi_consumer_1/subscription' \
--header 'Content-Type: application/vnd.kafka.v2+json' \
--data-raw '{
"topics": [
"suppliers"
]
}'
}'

Response

HTTP 204 Code: No Content

Get the records

Request

curl --location --request GET 'http://kafka-rest-cmn.test.tui-dx.com/consumers/sapfi/instances/sapfi_consumer_1/records' \
--header 'Accept: application/vnd.kafka.json.v2+json'

Response

[
  {
    "topic": "suppliers",
    "key": "1",
    "value": {
      "id": "1",
      "customer_code": "1",
      "telephone": "888582158",
      "email": "supplier1@test.com",
      "language": "EN"
    },
    "partition": 0,
    "offset": 3
  }
]

Conclusion

Thanks to Confluent Kafka Rest Proxy, the use of Apache kafka is not exclusive for languages or platforms with kafka clients supported, you can use it through API with easy development and installation as well as with community license.