How to build your Kafka Topology with JulieOps
An operational manager for Apache Kafka (Automation, GitOps, SelfService)
JulieOps, formally known as Kafka Topology Builder, is an open source project licensed under MIT License. It helps you automate the management of your Kafka Cluster handling topics, configurations and the required permissions. For further information refer to the docs.
Important configurations
Due to the different operations, Julie needs a global email account and password as well as cluster credentials (API Key and Secret). The cluster credentials must be from a global account (full access of the cluster) or a service account with access to the different resources that we want to be managed by Julie.
Julie manages a .cluster-state file, similarly as terraform manages a .tfstate file, which is read and written at the working directory when the Julie command is launched. For this to work properly, we must have the topology.state.topics.cluster.enabled and topology.state.cluster.enabled parameters set to false, otherwise Julie will not check the .cluster-state file and the changes will not be saved.
Structure of the project
Julie needs a common-config.properties file located at the root folder where all common parameters will be specified.
A project-config.properties file which can overwrite the configuration from the common file if needed, but its main purpose is to add the specific parameters of the project.
A topology.yaml file at the project level where the topology will be defined (topics, schemas, KSQL artefacts, managed connectors and principals).
A roles.yaml file at the project level to define the access policies ACL or RBAC. To enable the role deployment the parameter julieops.role at the project-config.properties must be pointing to the roles topology file path.
Common Configuration
In the common-config.properties file we can setup the common configuration for all the projects we have. This file will be located at the root folder of the project.
Some interesting parameters include topology.topic.prefix.format to define the full topic naming format, topology.topic.prefix.separator to indicate the separator when building topic names (by default is ‘.’), topology.acls.optimized to optimize ACLs in order to reduce the number of rules, allow/deny elements deletion with the parameters allow.delete.topics, allow.delete.principals, allow.delete.artefacts.connect, allow.delete.artefacts.ksql, parameters regarding the cluster-state file (remember set them to false) topology.state.topics.cluster.enabled, topology.state.cluster.enabled, the state management system (state processor class to store state) topology.builder.state.processor.class and the ones refering to the log4j properties.
To know more about all the important configuration values you can have a look at this section of the documentation.
Project Configuration
The project-config.properties file is located at each project folder and it aims to declare specific parameters for the project.
Here we can specify the confluent cloud environment, Kafka connection, schema registry URL, access control class, path to roles file and enable principals (use the Service Account ID internally for managing all required ACLs), example:
ccloud.environment=YOUR_CCLOUD_ENVIRONMENT# Kafka Connection
bootstrap.servers=PLAINTEXT://YOUR_CCLOUD_BROKER:9092
sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="YOUR_CCLOUD_API_KEY" \
password="YOUR_CCLOUD_API_SECRET";# Schema Registry URL
schema.registry.url=YOUR_SCHEMA_REGISTRY_URL# topology.builder.access.control.class=com.purbon.kafka.topology.roles.RBACProvider
topology.builder.access.control.class=com.purbon.kafka.topology.roles.SimpleAclsProvider# Roles -> julie.roles=/path/to/the/roles/file
julie.roles=roles.yaml# Use the Service Account ID internally for managing all required ACLs
topology.translation.principal.enabled=true
Topology
In the topology.yaml file, located at the same level as the project-config.properties, the Kafka topology of the project will be declared.
We are able to add any metadata we wish to our topology. An example of a simple topology with a topic, a producer and a consumer group:
---
context: "context-name"
projects:
- name: "project-name"
topics:
- name: "t-dom-example-test-name"
principal: "User:account_id1,account_id2"
config:
replication.factor: "4"
num.partitions: "3"
cleanup.policy: "delete"
delete.retention.ms: "2628002880"
metadata:
type: "t"
domain: "dom"
owner: "example"
name: "test-name" consumers:
- principal: "User:account_id"
group: "c-consumerGroup"
metadata:
type: "c"
anything: "value" producers:
- principal: "User:account_id"
metadata:
type: "p"
**Note: When defining a topic it is possible to use dataType to specify the data type of the topic or schemas if the desired behaviour is to register schemas for the topic.
Roles
In the roles.yaml file, located at the same level as the project-config.properties, the Kafka roles of the project will be declared.
An example matching the previous topology:
---
roles:
- name: "roles-name"
acls:
- resourceType: "Topic"
resourceName: "t-" # All topics starting by 't-'
patternType: "PREFIXED" # LITERAL or PREFIXED
host: "*"
operation: "READ,WRITE"
permissionType: "ALLOW" - resourceType: "Group"
resourceName: "c-consumerGroup" # Consumer group name
patternType: "LITERAL"
host: "*"
operation: "READ"
permissionType: "ALLOW"
** Note: The READ, WRITE, DELETE, ALTER permission types also imply the permission type DESCRIBE as said by Confluent.
Run Julie
Julie command:
julie-ops \
--clientConfig common-config.properties \
--overridingClientConfig project-config.properties \
--topology topology.yaml --brokers YOUR_CCLOUD_BROKER \
--dryRunNote that dryRun is used to print the execution plan without altering anything, it is optional.
This command will automatically create or update the .cluster-state file as well as creating or updating the topology in the Kafka Cluster.
Additional Resources:
Official GitHub repository: https://github.com/kafka-ops/julie
Julie Ops Documentation: https://julieops.readthedocs.io/en/3.x/index.html
Contribute to the project: https://github.com/kafka-ops/julie/blob/master/CONTRIBUTING.md
Do you like this content?
Subscribe to my medium page and be the first to get notified whenever I publish a new one!
Follow me on LinkedIn for daily insights about Software & Data Engineering 🫰🏻
