Introducing Azkarra Streams : The first micro-framework for Apache Kafka Streams

Florian Hussonnois
Nov 27, 2019 · 7 min read

Kafka Streams is a powerful library for building complex streaming applications on top of Apache Kafka. With time, and after multiple projects, we found ourselves writing same code for running and interacting with Kafka Streams applications in production.

We are convinced that the development of a simple micro-service based on Kafka Streams, with a minimum of features to get into production safely, should be a matter of days and not weeks.

So we decided to build our own framework in an effort to ease the development and the operation of Kafka Streams applications.

Today, we are excited to announce Azkarra Streams, a new open-source micro Java framework that lets you focus on writing Kafka Streams topologies code, not boilerplate code necessary for executing them.

Image for post
Image for post
Azkarra is Basque word for “fast”

Key Features

Azkarra Streams provides a set of features to quickly debug and build production-ready Kafka Streams applications. This includes, among other things:

  • Lifecycle management of Kafka Streams instances (no more ).
  • Easy externalization of Topology and Kafka Streams configurations (using Typesafe Config).
  • Embedded HTTP server for querying state store (Undertow).
  • HTTP endpoint to monitor streams application metrics (e.g : JSON, Prometheus).
  • Web UI for topologies visualization.
  • Encryption and Authentication with SSL or Basic Auth.
  • Etc.

Why Azkarra ?

Before writing your first Azkarra application, let’s take some time to describe the different parts of a standard Kafka Streams application in order to get a better understanding of Azkarra benefits.

For doing this, we are going to use the famous word-count example which is available on the official Kafka Streams documentation.

First of all, we have to declare and build a .

After that, we need to define the application configuration.

and to create the instance.

Finally, we have to manage the runtime part of the Kafka Streams application. This means, starting the instance and managing the clean shutdown of the application using a shutdown hook.

N.B : Here is the complete code : GitHub Gist.

Of course, a streaming application is rarely that simple. For example, you will need to add error handling and monitor the state of the Kafka Streams instance. In addition, you may wish to query internal stores using interactive queries, etc.

Last but not least, sometimes you have to deal with some issues (e.g: https://issues.apache.org/jira/browse/KAFKA-7380).

But the reality is that, as a Developer, you should always focus your development efforts on the definition and optimization of your topology. The reason is simple - this is the part that delivers value to your business.

First Steps with Azkarra

One of the first aspects we wished to resolve was the separation of concerns between the build of and its execution. Indeed, we thought that creating and starting a new instance should not be managed directly by developers.

So, let’s rewrite the WordCount example using the Azkarra API.

First, we are going to use a Azkarra Streams Maven Archetype for creating a simple project structure. You can run this following command :

The already contains the Azkarra Streams and Kafka Streams dependencies :

Using your favorite IDE or editor, open the Maven project. For this introduction, we will remove the bundle example and start our first app from scratch.

After that, let’s create a new file under directory , with a basic java method and our topology definition as follows :

As you can see, we have only implement a interface named in order to build the . Azkarra Streams is responsible for automatically configuring and starting the so-called instance.

Next, we have to configure our application. We will create a simple file named in to contain the following code :

N.B : You can also leave the file already present in the project as it is.

Congratulation! You just write your first streaming application using Azkarra.

Running Your App on Docker

For executing our application, we have to first start a Kafka Cluster. For that purpose we will use the official Kafka Docker images maintain by Confluent.Inc.

To start a single-node Kafka Cluster run the file contains in the project.

Then, create the two topics (source, sink) used by the topology. For that, you can run the provided script :

As a last step, we will package and run the Maven project :

To verify, your streams application is running, check the health endpoint :

Finally, let’s produce some messages to Kafka topic :

To consume output topic :

Embedded HTTP Server

One of the key features of Azkarra, is the embedded web-server that exposes endpoints for managing and monitoring local streams applications

For example, you can list the stream instances running locally (i.e : the instances executed in the JVM).

You can also get information about a specific local streams instance.

Finally, you can export the streams application metrics into Prometheus format :

Azkarra Streams exposes additional endpoints : For more information, have a look to the REST API Reference.

Azkarra WebUI

Another cool feature of Azkarra Streams, is the default embeded Web UI available on : http://localhost:8080/ui that let you manage your streams application.

Image for post
Image for post
Azkarra WebUI — Overview

Azkarra WebUI was first designed to facilitate development but has quickly evolved into a mini-administration interface.

For example, you can stop, restart a streams application using the “Available Actions” button, explore metrics, configuration and so on.

Azkarra WebUI also ships with a simple DAG representation for the streams topology.

Image for post
Image for post
Azkarra WebUI — Topology DAG

Interactive Queries

Finally, Kafka Streams has a great mechanism to query the states materialized by streams applications. Usually, as developers we build HTTP endpoints to expose these states using the public Kafka Streams API.

Azkarra Streams provides a default endpoint for that purpose which is accessible directly through the Azkarra WebUI.

The digram below, shows a query ‘all’ on the state store ‘WordCount’

Image for post
Image for post
Azkarra WebUI — Interactive Queries — All

Under the hood, the Web UI runs the following HTTP request :

Here is another example to only query a specified key.

Image for post
Image for post
Azkarra WebUI — Interactive Queries — Get Key

Going further

If you want to read more about using Azkarra Streams, the documentation can be found on GitHub Page. The documentation contains a step by step getting started to learn basic concepts of Azkarra.

The project also contains some examples.

Conclusion

Azkarra Streams is an initiative to enrich the Kafka Streams ecosystem and facilitate its adoption by developers through a lightweight micro-framework.

We hope this project will be well received by the open-source and Kafka community. Azkarra is still evolving and some features need improvements.

To support Azkarra Streams project, please ⭐ the Github repository or Tweet if this project helps you!

Thank you very much!

About Us :

StreamThoughts is an open source technology consulting company. Our mission is to inspire companies to create ever more innovative services that make the most of the opportunities offered by real-time data streaming.

We deliver high-quality professional services and training, in France, on the Apache Kafka ecosystem and Confluent.Inc Streaming platform.

StreamThoughts

Enable developers to be more impactful and successful through open-source technologies

Florian Hussonnois

Written by

Co-founder @Streamthoughts , Apache Kafka evangelist & Passionate Data Streaming Engineer, Confluent Kafka Community Catalyst.

StreamThoughts

We are a small team of experts. By helping our customers to make values out of their data as real-time event streams through our expertise, solutions and partners, we open up unprecedented possibilities for them to innovate, evolve and adapt to their future business challenges.

Florian Hussonnois

Written by

Co-founder @Streamthoughts , Apache Kafka evangelist & Passionate Data Streaming Engineer, Confluent Kafka Community Catalyst.

StreamThoughts

We are a small team of experts. By helping our customers to make values out of their data as real-time event streams through our expertise, solutions and partners, we open up unprecedented possibilities for them to innovate, evolve and adapt to their future business challenges.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store