What’s New in Azkarra Streams v0.5

Florian Hussonnois
Jan 7 · 4 min read
Create Kafka Streams applications faster than ever before!

I’m the pleasure to announce the release of Azkarra Streams 0.5

This release includes not only significant new features but also a few breaking changes in public APIs.

When I started the Azkarra project it was just a few utility classes used to wrap the Kafka Streams and Topology classes. But, the project has quickly evolved to become a more elaborate development framework. Since its first open-source release, we got feedback from the community on how the project starts to be used and which features could be improved. So I thought it was a good time to refactor some internal APIs since we are still in the early stages of the project.

Azkarra Worker

Azkarra Streams 0.5 introduces a new module called azkarra-worker that is used to build and run a standalone Azkarra worker installation.

The basic idea behind Azkarra Worker is inspired by Kafka Connect. In many situations, you may want to share and reuse some Kafka Streams topologies across your organization. For example, it may be a KafkaStreams topology to convert messages between different formats (e.g., JSON to Avro) or a topology to compute the number of messages having a field equals to a given value, etc.

Azkarra Worker provides the default main Java class which is necessary to bootstrap an application. Furthermore, it contains the scripts used to start and stop the application.

External components

Components, like TopologyProvider, can now be loaded from directories external to the java classpath. The new azkarra.component.paths property can be configured with a list of directories (separated by a comma) to indicate the locations from which the components will be scanned.

Azkarra follows the same logic used by Kafka Connect for loading connector-plugins. Thus, each configured directories may contain:

  • an uber JAR containing all of the classes and third-party dependencies for the component (e.g., topology).

Classloading Isolation

For each external component, Azkarra will use a dedicated Java ClassLoader. On the one hand, this allows avoiding library version conflicts between components, and, on the other hand, this allows running multiple versions of the same streams topologies.

Azkarra Docker Image

A Docker image is now available on Docker Hub for easily run Azkarra Worker :

$> docker run --net host streamthoughts/azkarra-streams-worker
--mount type=bind,src=/tmp/azkarra/application.conf,dst=/etc/azkarra/azkarra.conf \
--mount type=bind,src=/tmp/components,dst=/usr/share/azkarra-components/ \
streamthoughts/azkarra-streams-worker

Note that the Azkarra docker image doesn’t contain any topology. So you have to mount the volume containing your external components to load.

The project contains a simple docker-compose example.


Streams Lifecycle Interceptor

This version adds the new pluggable interface called StreamsLifecycleInterceptor that allows you to execute operations before and after starting or stopping a Kafka Streams instance.

Azkarra provides two built-in interceptors for common operations:

  • AutoCreateTopicsInterceptor: Azkarra can now be configured to automatically create the source and sink topics declared by the topology before starting the KafkaStreams instance. This interceptor can also be used for deleting topics after the instance is stopped (only for development).

Documentation : The StreamsLifeCycle Intercepting Chain


Kafka Streams Factory

In the previous Azkarra version, the KafkaStreams instance was built internally. But in some cases, you may want to be able to customize how the KafkaStreams instances are created. For example, it may be to provide a KafkaClientSupplier that will add some tracing mechanisms on top of the Kafka clients (e.g., kafka-opentracing).

Azkarra API now exposes the new KafkaStreamsFactory interface that you can implement to build your instance.

The following is a simple example demonstrates on to register a KafkaStreamsFactory:

StreamsExecutionEnvironment env = DefaultStreamsExecutionEnvironment.create();env.setKafkaStreamsFactory(() -> new KafkaStreamsFactory() {
@Override
public KafkaStreams make(Topology topology,
Conf streamsConfig) {
KafkaClientSupplier clientSupplier = //...
return new KafkaStreams(
topology,
streamsConfig.getConfAsProperties(), clientSupplier);
}
})

Documentation : Kafka Streams Factory


Components Definition

Azkarra provides a minimal mechanism for declaring components and enabling dependency injection. This mechanism has been the subject of many refactoring tasks to provide more functionalities.

Component Factories

Azkarra supports the new Factory annotation which can be used for annotating a class that provides one or many components defined by methods annotated with Component.

The following is a simple example:

@Factory
public class TopicsFactory {

@Component
@Singleton
public NewTopic streamsInputTopic() {
return new NewTopic(
"streams-plaintext-input", 6, (short)1);
}

@Component
@Singleton
public NewTopic streamsOuputTopic() {
return new NewTopic(
"streams-wordcount-output", 6, (short)1);
}
}

JSR-330

Azkarra now leverages some of the Java annotations defined by the JSR-330 (javax.inject) — Dependency Injection for Java specification :

However, you should note that Azkarra doesn’t support theInject annotation.

Restricted Qualifier

Ths release add a new Restricted annotation that can be used to limit/qualify the scope of use of a component either by environment or streams topology.

In the example below the CustomStreamsFactory will only be used for the streams topology named wordCountTopology.

@Component
@Restricted(type = Restriction.TYPE_STREAMS,
names = "wordCountTopology"
)
public class CustomStreamsFactory implements KafkaStreamsFactory {

@Override
public KafkaStreams make(Topology topology,
Conf streamsConfig) {
return new KafkaStreams(
topology,
streamsConfig.getConfAsProperties()
);
}
}

Documentation : Dependency Injection



About Us :

StreamThoughts is an open-source technology consulting company. Our mission is to inspire companies to create ever more innovative services that make the most of the opportunities offered by real-time data streaming.

We deliver high-quality professional services and training, in France, on the Apache Kafka ecosystem and Confluent.Inc Streaming platform.

StreamThoughts

StreamThoughts is an open source technology consulting company. Our mission is to inspire companies to create ever more innovative services that make the most of the opportunities offered by real-time data streaming.

Florian Hussonnois

Written by

Co-founder @Streamthoughts , Apache Kafka evangelist & Passionate Data Streaming Engineer, Confluent Kafka Community Catalyst.

StreamThoughts

StreamThoughts is an open source technology consulting company. Our mission is to inspire companies to create ever more innovative services that make the most of the opportunities offered by real-time data streaming.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade