Demystifying Apache Kafka with Kalibrate
Apache Kafka
Since its advent into open-sourcing in 2011, Apache Kafka has become a quietly influential part of an interconnected world. Able to dependably process millions of messages per second and at scale, Kafka works under the hood of 80% of Fortune 100 companies and countless other enterprises. If you’ve never heard of Kafka, you might be surprised to learn, for example, that your most recent visit to LinkedIn, or the last Uber you hailed, were brought to you by this platform.
If you have heard of Kafka, then you probably already know that it’s popular — and you probably also know that it’s complicated. To research Kafka is to dive into a world of acks, leadership elections, and hierarchical timing wheels. For some, this complexity is its own reward, but many users want the utility of Kafka without the complication. Cloud service providers like Confluent and Amazon will abstract away these difficulties — for a price, of course.
Introducing Kalibrate
The expense and obscurity of existing Kafka management solutions inspired us to create Kalibrate. Kalibrate is a free and open-source Kafka management web application that provides cluster management and monitoring capabilities in a sleek, secure, and easy-to-use format.
Kalibrate connects to your cloud-hosted or locally run Kafka clusters with a simple URI and, if applicable, SASL credentials. Once connected, you can:
- View metadata for cluster brokers, topics, messages, partitions, groups, and consumers
- Access dynamically updated visualizations of cluster throughput, offsets, and replica health
- Configure cluster elements and export data in a CSV format
Kalibrate remembers your connected clusters between sessions, symmetrically encrypting any sensitive credentials, so that you can conveniently check back in on your clusters. Kalibrate supports optional in-app alerts and, for when you’re away, Slack and email notifications. Kalibrate was developed with user experience in mind, and features light and dark modes, as well as user account configurations.
If you have no Kafka clusters of your own — or want a low-stakes example to play with — our GitHub repository includes a dockerized e-commerce demo cluster that you can run locally. Check out the readme for more details.
“One learns at all costs” — Franz Kafka
We — like any good Kafka initiates — were drawn to the nuances of the platform. We learned the nuances of Kafka and its JavaScript client, KafkaJS. We resolved to build our own time-series graphing functionality, which involved careful management of React state, long polling, and graphic libraries. Our user database presented some points of interest — in addition to normal security considerations like JWTs and hashing passwords, we implemented two-way encryption to secure users’ cloud credentials while allowing them to be accessed in subsequent sessions. We carefully weighed the convenience of component libraries against the challenges they present in terms of customization and application size. It’s hardly groundbreaking in 2023 to sing the praises of type safety, but this project is a perfect use case for it: the guardrails provided by Typescript and Prisma saved us countless hours of debugging as we wrangled myriad API calls and complex response bodies. We discovered share-worthy Kafka media like this video and this picture book that, like Kalibrate, go a long way toward making Kafka approachable.
Getting Started
Make your Apache Kafka cluster management and monitoring a breeze by heading on over to Kalibrate and setting up a user account.
If you prefer to run the application locally, you can hit the ground running without much more effort by following these instructions:
- Clone the repository from GitHub
- In your terminal, run:
npm run docker-build-dev
npm run docker-dev:up3. Navigate to localhost:5712 to access the application.
Thanks for reading!
Apache Kafka is a powerful tool, but reaping its benefits doesn’t have to be complicated. We built Kalibrate to make working with Apache Kafka an approachable and intuitive experience. Try Kalibrate out, or contribute to it, by visiting our website or GitHub, and keep up with the latest by following Kalibrate on LinkedIn and Twitter.
Questions? Feature ideas? We’d love to connect!
Jonah Durning-Hammond | GitHub LinkedIn
Software Engineer based in New York
Ashlee Gafaru | GitHub LinkedIn
Software Engineer based in New York
Jihui Xue | GitHub LinkedIn
Software Engineer based in New York
Rebecca Kwong | GitHub LinkedIn
Software Engineer based in New York
Julien Devlin | GitHub LinkedIn
Software Engineer based in New York
