TLDR Kafka Wakes Up And Is Metamorphosed Into A Database by TP Morgan

They added KQL — one more step towards a complete “database inside out” solution.

• Databases were like set of tablets, but now they are more like ever changing data-processing machines on top of data stream.
• So Confluent created Kafka — a materialized data stream. And they build it into a a platform in its own right — by adding KQL for example.

We live in a topsy-turvy world where databases are trying to add streaming features and streaming servers are becoming databases.

• The purpose of the Kafka’s SQL interface — to lower the barrier to entry to using streaming services based on Kafka, much as SQL overlays on top of Hadoop are meant to make it easier for regular SQL users.
• The KSQL overlay supports a variety of streaming functions, such as windowed aggregations to stream table joins, and it is currently in developer preview.

Why would you put a database into the streaming layer?

The relational database has a transaction log, which is the source of truth, and it has tables that are derived from this log that are the first order construct. With KSQL, the log is a first order construct and tables are derived views.

• KQL is not the only one — Apache Flink has an SQL engine based on the Apache Calcite. PipelineDB is a streaming layer that runs on PostgreSQL and Vertica databases. Spark Streaming has DataFrame and SQL layer.
• KQL is not ANSI SQL compliant due to the streaming nature. So you can’t just use your favorite SQL driver.
• Confluent co-founder Narkhede says they wants to spend time getting feedback on KQL from the Kafka community to see how to tweak it and then commercialize.

TLDR — It is not ready for primetime yet.

This week sponsor okmeter.io — infrastructure and kafka monitoring.