MASH by Envoya: The New Grafana plugins offering Management and Visualization for Kafka Clusters
Messaging Queues Lack a Visual Monitoring Solution
Many companies are transitioning from a monolithic to a microservice architecture application, creating a need for messaging queues like Apache Kafka. Kafka’s publish-subscribe system is highly complex and it’s critical to have a tool that can simultaneously help developers visualize and monitor message passing between microservices applications. Kafka is widely used in more than one-third of Fortune 500 companies to provide real-time data analytics. Kafka at this moment lacks a good monitoring solution and does not offer developers a visual platform to manage their message queuing systems.
MASH is a Visualization and Monitoring Tool for Kafka Clusters
MASH is a series of Grafana plugins designed to work specifically with Kafka to help developers visualize, monitor and handle the flow of messages between their microservices on a single platform. Grafana is an open-source visualization tool, popular in the DevOps monitoring space where MASH was developed to make it easy for developers to incorporate each plugin into their current Grafana dashboard. MASH displays core metrics and it was designed to specifically visualize Kafka clusters. MASH brings you a novel way to simply understand the health of your consumers, and show metrics such as consumer lag in a way that can be understood instantly. Data is easily formatted and organized to display average consumer rates and outlier message consumption rates. MASH helps developers to have a visual aspect and a monitoring component without having to deal with the complexity of creating a custom panel.
MASH Generates Mock Data and Includes a Kafka Cluster
In addition to the visualization of these critical metrics, MASH also provides an integrated testbed for message production and consumption to an existing or test Kafka cluster. During the development of MASH, the engineering team at Envoya found a need for a data generator that could produce mock data to the Kafka platform. Therefore, MASH incorporates a full testbed, as it could be integral for other developers to use for testing their current Kafka cluster. In addition, a Kafka cluster is provided in the testbed where developers can tune existing or create new microservice architectures.
MASH Works Easily with Kafka Exporter, Prometheus, Grafana, and Docker-compose
MASH is integrated tightly into the tools you already use. MASH uses the Grafana platform to present its panels and dashboards. In addition, MASH uses Prometheus with the Kafka Exporter for data collection and aggregation. MASH will need a running instance of Kafka, along with Kafka Exporter and Prometheus. If you are not currently integrated with these tools a docker-compose file has been provided for your convenience and can be initialized with a single command line in your terminal.
MASH IS AVAILABLE TODAY! (08/06/20)
Mash is available for development purposes on the MASH’s GitHub and is being submitted for full integration into the Grafana plugin store, for use with your existing or future dashboards and monitoring environments!
Feel free to follow the directions provided in our GitHub to create a local environment. The team at Envoya invites all testing, feedback, and suggestions that the community can provide. We pride ourselves on our commitment to the Open Source Community and believe in the collaborative nature of software design and growth.
Thank you to the amazing developing team at Envoya: Steve Budarz, Besik Rurua, Victoria Balandeu, and Cynthia Franqui.
