kafkaVision: An open-source monitoring tool for Apache Kafka
In the modern era, data is everything. It’s no surprise that a huge number of technologies and frameworks have been developed over the last decade to help accumulate and manage the data that drives companies, their products, and their business decisions. Apache Kafka is one such technology.
What is Apache Kafka?
Originally created in 2011 by LinkedIn to more efficiently track user behavior with real-time event processing, Apache Kafka has since evolved into a robust distributed messaging system used by leading companies like Netflix, Shopify, Uber, and GrubHub.
Kafka’s functionality can be summarized in three points:
- It creates a data pipeline between event publishers and event subscribers
- It stores records in these pipelines sequentially and reliably
- It processes the stream of records in real-time
Developer Woes
The swift rise in popularity of Kafka means more and more developers are being charged with optimizing and monitoring the health of Apache Kafka applications. Unfortunately, Kafka does not provide a native visualization tool that would allow devs to observe the status of their cluster at a glance. Enter kafkaVision.
Introducing kafkaVision
kafkaVision is a free-to-use, open-source monitoring tool developed under the tech accelerator OSLabs. kafkaVision delivers a modern, intuitive GUI that provides a low-latency monitoring solution for the most essential health metrics of a Kafka cluster. In addition, kafkaVision presents simplified Topic, Partition, and Offset data to help developers make optimization decisions about partitions and consumers on the fly.
To use kafkaVision, you can choose to take advantage of the health metric graphs and/or the topic visualizer by connecting your existing (and running) Kafka application.
How kafkaVision saves developers time:
- Prioritizes cluster health metrics and status updates that really matter
- Provides relevant and useful graphs that update with low latency — no additional configuration required.
- Organizes topics and partitions in an easy-to-consume visualization to inform quick consumer optimization decisions
- Eliminates the need for manual creation of PromQL queries and unfamiliar CLI commands
Getting Started
kafkaVision is currently in beta and the development team is committed to incorporating user feedback into our plans for future development. If you employ Apache Kafka in your application, we encourage you to clone the kafkaVision GitHub repo and give it a spin.
Connect with the kafkaVision team
kafkaVision || GitHub — LinkedIn
Isabelle Salvador || GitHub — LinkedIn
Kayliegh Hill || GitHub — LinkedIn
Neel Lakshman || GitHub — LinkedIn
Rob Beier || GitHub — LinkedIn
Support the Project
If you’d like to support the active development of kafkaVision and OSLabs:
- Clap this Article!
- Follow kafkaVision on LinkedIn for future updates
- Fork or star the repo on GitHub
- Contribute to this project by raising a new issue or making a PR to solve an issue
