Kafka Topic Naming Conventions: Best Practices
Apache Kafka is a popular distributed streaming platform that thousands of companies around the world use to build scalable, high-throughput, real-time streaming systems. One of the most controversial and hot discussions around this technology for years has been the Kafka Topic Naming Conventions. In this post, I will provide the best practices on how to name Kafka topics. I will guide you on deciding the naming conventions and the do’s and dont’s when you setup your systems.
Note: Originally published for SocketDaddy.com
Despite its popularity among companies like The New York times, Pinterest and LinkedIn, there’s little guidance on naming Kafka Topics. There is of course tons of material on how to decide the partitions and replication factor for your topics. However, there is not much information on how to name them.
You typically would decide the names for the topics based on conventions and practices followed at your company. These could also be based on your personal preferences. If you look at the very few results that show up on Google for Kafka Topic Naming Conventions, most results would recommend a convention that would look something like these:
<namespace> .<product>.<event-type>
<application>-<data-type>-<event-type>
<organization>.<application-name>.<event-type>.<event>
Chris Riccomini in his excellent blog post says that he has had great success with the following convention:
<message type>.<dataset name>.<data name>
All of these look perfectly fine and meet the needs. However, is there more than what meets the eye? Lets take a more detailed look at what makes a topic name a good one.
What names are valid in Kafka?
Kafka enforces a set of “legal” characters that can constitute a topic name. Valid characters for Kafka topics are the ASCII Alphanumeric characters, ‘.’, ‘_’, and ‘-‘. So, anything that matches the following pattern can be a valid Kafka Topics name.
val legalChars = "[a-zA-Z0-9\\._\\-]"
However, one thing to keep in mind is that due to limitations in metric names, topics with a period (‘.’) or underscore (‘_’) could collide. To avoid issues it is best to use either, but not both.
There is no control or way around what Kafka enforces. However, you can use this rule as a foundation to get creative and come up with more standard naming conventions.
It is worth emphasizing that the topic names are case sensitive. So topicName is not the same as topicname or TopicName. Kafka would treat all three of them as individual topics.
Naming the topics
Lets look at some guidelines that you should consider when naming your Kafka Topics.
Decide the format for the topic names
The first and important thing you need to consider is the format you would want to follow for all your topics. As we mentioned before, Kafka allows all ASCII Alphanumeric characters, periods, underscores and hyphens. You can format a topic name in several ways. Some examples are:
my-topic-namemyTopicNamemy_topic_nameitlabs.mytopic.nameITLabs-Website-Tracker
It is extremely important to decide this as a first step because you need to have your naming conventions and pattern consistent. Readability and ease of understanding the topic names play a key role in these decisions and you don’t want to have inconsistent patterns in the topic names your system.
What fields should be part of your Kafka topic naming conventions?
The next step in naming Kafka topics is defining what fields should go in the name and in what order should they appear. Some of the best practices Chris Riccomini suggests are:
Do not use fields that change.
Avoid fields in the topic names that would change over time — such as the consumer name, the team names, the owner of the topics, etc. Once you create a topic in Kafka, it is impossible to rename them.
Leave metadata and schema information out of names
If you can find the nature of the data in the topic or the information related to the field elsewhere, such as the Kafka Metadata or a schema registry, leave them out of the topic names. The schema registry can provide you with the information about a schema for a given topic. Kafka brokers provide topic metadata information. Since there are other sources of truth for this information, its best to keep them out of the topic names.
Avoid Partitions, security info etc in topic names
This is similar to the previous point. All the metadata information such as the partition count, the security levels, configurations are available in the topic’s metadata and through the Kafka Brokers. You should avoid including these fields in topic names.
Don’t tie topic names to consumers or producers
You should never decide a topic name based on the producers or the consumers of that topic. The number of producers and the number of consumers can change over time. You should not include in a Topic-name a field value that is dynamic and changes over time .
Enforcing Naming Conventions
The first step to make sure that the users adhere to the naming conventions is to disallow any random user from being able to create topic. You can do this by disabling the Auto-create Topic functionality in Apache Kafka by setting auto.create.topics.enable=false in the broker configurations.
Another way to ensure the naming conventions are followed is to automate the process of creating the topics and the fields used the create the topic names are taken as inputs.
There should also be a automated script or an utility that monitors the topics in a Kafka Cluster to validate the topic names and flag any violations of the standard naming conventions and formats.
Conclusion
While you can technically name your Kafka Topics anything you want (as long as it meets the Kafka Legal Character rules), it is certainly important that you have a standard naming convention for the Kafka topics you create. This is essential to make sure that your Kafka environment does not become cluttered. It is important that such naming conventions and standards are enforced earlier on in the environment because once you create a Kafka topic, it is impossible to rename them.
References:
