Kafka Observability

Photo by NASA on Unsplash

Who doesn’t remember the film Apollo 13, a classic from the 90' and the quote “Houston, we have a problem” ?

I recently worked with Kafka; I could not find a mature tech article to explain step-by-step what to do to have AiOps & Observability for Kafka, including alerting, automation, and deep analytics, without having complex monitoring architecture in place.

This article will explain how to create a small demo Kafka cluster and get metrics into Splunk Observability smartly and without exposing the Java JMX port.

Example architecture Cluster Operator

Kubernetes Cluster overview

I am using just one node minikube cluster, installed using this cmd:

minikube start --cpu 4 --memory 6192 --container-runtime=cri-o --disk-size=60G --network-plugin=cni --cni=bridge

minikube profile list

Installation

Those are five main parts that I will cover in the next points:

1. Install Helm on your system;
2. Install Strimzi Cluster…