Getting Started with Ballerina Kafka

Ballerina is now ships with Kafka module! This setup is no more needed!

Ballerina is a rising, cloud-native programming language. Even though there are many languages out there, ballerina is one of a kind.

Kafka, on the other hand is a rising, well-used distributed streaming platform. It is widely used in messaging domain nowadays.

Now, what is WSO2 Ballerina Kafka ?

The WSO2 Kafka module is the set of Ballerina APIs for using apache Kafka. If you’re developing a system, which uses Kafka as your messaging platform, and you’re thinking of developing the system using Ballerina, here’s what you need !

If you’re new to Ballerina, please take sometime to learn about Ballerina, before reading this any further. Ballerina has a vast range of ballerina examples to start with. You can find the Ballerina API Documentation here.

wso2-kafka module provides user a range of APIs. You can make a Kafka Producer using Ballerina, and can publish to a Kafka topic right away! Or you can create a Kafka Consumer, subscribe it to a topic and retrieve data.

But the best part is, the Service model !

Ballerina is using service-listener model, in almost all their libraries; wso2-kafka is no exception. You can create a Kafka Listener using wso2-kafka module, and listen to certain Kafka topic(s). When someone publishes to a topic to which a Listener is listening, it will retrieve those data, right away!

There are many use cases we can consider using wso2-kafka module. But first of all, I am going to explain how to setup wso2-kafka module to function correctly.

Setting up the Environment

First of all, you need to download Ballerina distribution. The latest version as of the day this is written is Ballerina V0.990.2. You can download ballerina from here.

Then you need to download Apache Kafka, V1.0.0 or higher. (Ballerina Kafka connector supports to Kafka Versions 1.x and 2.0.x as of now.). You can download the Apache Kafka from here.

Once you download these two, you have to download wso2-kafka connector. You can download wso2-kafka connector from here. After downloading you can install your wso2-kafka module using following ways:

Method 1: Using installation scripts

Unzip the wso2-kafka-<version>.zip and go into the extracted directory. Inside that directory you can find two scripts:install.sh and install.bat. Use the one compatible with you ( install.sh for Linux and Mac, install.bat for windows.)

Provide the BALLERINA_HOME location (which you downloaded before) when asked. This will automatically install wso2-kafka module to your ballerina distribution.

If your installation is successful, following message will appear in your terminal.

Please enter Ballerina home: <YOUR_BALLERINA_HOME>
Successfully installed Kafka module!

Method 2: Manually Installing Ballerina Kafka Module

Unzip the wso2-kafka-<version>.zip and go into the extracted directory. Then copy the balo/wso2/kafka directory, into the <BALLERINA_HOME>/lib/repo/wso2/ directory. Now you should have a directory named kafka inside the <BALLERINA_HOME>/lib/repo/wso2/ directory.

Then go back to the wso2-kafka directory, and copy dependencies/wso2-kafka-module-<version>.jar file into <BALLERINA_HOME>/bre/lib/ directory.

From either of aforementioned methods, you are installing wso2-kafka module into your Ballerina distribution. Now we’re all set for developing our application.

Uninstalling WSO2-Kafka Module from Ballerina

If you want to uninstall wso2-kafka module, there are two methods, as similar to the installation methods, mentioned above.

Method 1: Using Uninstallation Scripts

Go inside your wso2-kafka directory. There are two uninstall scripts; uninstall.sh and uninstall.bat. If you’re a Linux or a Mac user, use uninstall.sh. If you’re a Windows user, use uninstall.bat. Provide the correct BALLERINA_HOME when asked.

If the uninstallation is successful, your command prompt will show the following message:

Enter Ballerina home: <YOUR_BALLERINA_HOME>
Successfully uninstalled Kafka module!

Method 2: Manually Removing Ballerina Kafka Module

You can manually uninstall wso2-kafka module. To do so, delete the following file:

<BALLERINA_HOME>/bre/lib/wso2-kafka-<version>.jar

You also have to delete the following directory:

<BALLERINA_HOME>/lib/repo/wso2/kafka

Now the wso2-kafka module is successfully uninstalled.

Note: You can’t have two wso2-kafka versions in the same time. If you didn't uninstall the kafka module, before installing the newer version, it might cause you problems.

In our next article, let’s see how to implement a sample review filtering system using wso2-kafka.

Note: Apache Kafka and Ballerina logos are trademarks of the respective companies.