Kafka on Kubernetes? Confluent Has Made It

Simon Sugob

Published in

HireDevOps

4 min readNov 8, 2018

Easily Deploy Kafka on Kubernetes with Confluent Operator

1. Kafka on Kubernetes — What They Are

What Is Apache Kafka?

“Apache Kafka® is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Its storage layer is essentially a “massively scalable pub/sub message queue designed as a distributed transaction log,” making it highly valuable for enterprise infrastructures to process streaming data”<Wikipedia>.

What Is Kubernetes?

“Kubernetes is an open-source container-orchestration system for automating deployment, scaling and management of containerized applications. (…) It aims to provide a “platform for automating deployment, scaling, and operations of application containers across clusters of hosts”. It works with a range of container tools, including Docker“<Wikipedia>.

2. Deploying Kafka On Kubernetes NOW Easier With Confluent Operator

The Confluent Operator

The widespread use of Kubernetes by the innumerable number of platforms in the company gave rise to Confluent Operator — says Neha Narkhede, Co-founder and CTO at Confluent — “the best way to deploy Apache Kafka and Confluent Platform on Kubernetes“. Alex Woodie rightly noticed that “many organizations have attempted to wire Kafka to run on the Kubernetes cluster manager, but it has been a challenging technical feat that Confluent has succeeded at!”

“Enabling everyone to run Apache Kafka® on Kubernetes is an important part of our mission to put a streaming platform at the heart of every company. This is why we look forward to releasing an implementation of the Kubernetes Operator API for automated provisioning, management, and operations of Kafka on Kubernetes” <Neha Narkhede>

“This includes a rollout of pre-built Docker images for the entire Confluent Platform, Kubernetes deployment templates, a reference architecture that enables users to deploy Kafka and Confluent Platform on Kubernetes and an implementation of the Kubernetes Operator API for automated provisioning, management, and operations of Apache Kafka on Kubernetes” — adds Confluent’s CTO.

Kafka + Kubernetes expertise = The Confluent Operator

Rohit Bakhshi, the author of the article “Getting Started with Apache Kafka and Kubernetes” best sums up this solution:

“Our goal is to make streaming data ubiquitous: Kubernetes lets you run your apps and services anywhere, Kafka enables you to make your data accessible instantaneously, anywhere”.

Despite the fact that the link between Apache Kafka on Kubernetes may seem like a ‘match in big data heaven’, the devil is in the details. What were the biggest obstacles to overcome? Read more.

3. Why Should Your Deploy Kafka On Kubernetes With Confluent Operator?

Neha Narkhede explains that “the Confluent Operator directly addresses the challenges of running Apache Kafka on Kubernetes, and will offer the following features across all Confluent Platform components”:

Automated Provisioning
Cluster Management and Operations
Resiliency
Monitoring

[Read more]

4. Getting Started — How To Easily Run Apache Kafka on Kubernetes

Confluent provides two resources to install Kafka on Kubernetes:

▶Helm Charts — a starting point for deployment of Confluent Platform components

▶Recommendations for Deploying Apache Kafka on Kubernetes — a white paper

💡TIP: Not only I advise you to get familiar with these tools but also recommend you to have a look at two below resources to fully understand the deployment part:

Stateful, Stateless and Serverless — Running Apache Kafka on Kubernetes — online talk featuring Joe Beda, Heptio CTO and Kubernetes Co-creator, and Gwen Shapira, Principal Data Architect at Confluent.
Apache Kafka on Kubernetes — Could you? Should you? — this article gives the first reflection after the launch of Helm Charts.

You can already Sign up to Orchestrate Apache Kafka®on Kubernetes