Tools for Event-Driven DevOps with Apache Kafka

Image for post
Image for post
“Empty highway road in through a colorful desert” by Johannes Plenio on Unsplash

DevOps is a long road; it is a journey. Are you Dev or Ops or both? This is the challenge I have struggled with as DevOps has become a thing. I’m part-way through writing a Kafka DevOps blog post for Confluent and this thought is always at the top of my mind. However, while I grapple with it — I figured I would provide a compiled list of Kafka DevOps tools we see in the wild.

An important point, much like the DevOps definition below, is that Kafka DevOps is not clearly defined; it is what you make of it. Each organization will have its own predispositions and DevOps culture. However, to properly embrace the event-driven architecture, there is a fundamental upheaval of existing concepts. Teams and functionality need to be structured differently, aligned to deliver business functionality, based upon dataflows and related bounded contexts. When one of the key benefits is a 24x7 run-time, and dynamic path execution (dataflow routing); a new feature is enabled using a feature flag; then we are winning. We can avoid the dreaded big bang release that fills you with fear of having to spend the rest of the weekend doing a rollback.

Image for post
Image for post
Some streaming applications can get complicated. Proper testing and ops strategies are essential

Kafka Specific

KSQL

Infrastructure

Schema Registry integration

DataOps/generation

  • KafkaCat generic non-JVM producer and consumer for Apache Kafka; think netcat for Kafka — by Magnus
  • Kaf (like KafkaCat — Kafka CLI inspired by kubectl & docker)
  • KSQL (Community license)
  • Lenses SQL
  • EventSim for data generation of simulated data
  • Kafka Avro datagen simulation tool

General

What’s missing

Quite a lot of stuff is missing, but that’s because most environments will be geared towards a particular deployment and automation runtime.

  • Data tooling — topic provisioning, topic pre-population (from source or flat file), topic-assertion-validation
  • Infrastructure automation tools (Ansible, Helm, Helm charts, Operators)
  • Schema/quota/governance automation via APIs and assertions
  • Security/ACL automation (Very important)

Resources

Want to learn more?

@Liquidlabs, ex Confluent, CTO-Excelian, Thoughtworks. Founder Logscape, Startups, code-monkey, songwriter tennis-nut, snowboarding-nut, and Dad!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store