Stream Processing 101: From SQL to Streaming SQL in 10 Minutes
If i can query data at rest with SQL, why can’t I query streaming data with SQL?
Long drawn fight between SQL and NoSQL is drawing to the end. NoSQL databases do not have the momentum they once had and settling to 100–300m range revenue, a mere drop in 30 billion database market. Most importantly, NoSQL database are embracing SQL as the programming API.
Meanwhile “Streaming SQL”, a SQL like language that can query data streams as they come in, is rising. In years 2016–2017, we saw almost all open source stream processors, Storm, Kafka Streams, Flink, Spark Streaming, and Samza adding support for “Streaming SQL”. Stream processing finally converge under venerable SQL flag, which is familiar, simple, yet powerful way to query data.
To answer the question we started with, yes you can. Given the momentum, and the fact that everything converging to a known and tested SQL suggests that this trend is here to stay. Complex Event systems, the older incarnation of the same tools had SQL for years.
Goal of this post is to introduce the article “Stream Processing 101: From SQL to Streaming SQL in 10 Minutes”. The article introduces Streaming SQL, why it is needed, and its basic mechanisms using examples from Siddhi SQL and Kafka KSQL streaming SQL languages. Checkout the article to understand Streaming SQL, it’s rise, and it’s underpinnings.
Hope it is useful.
- http://flink.apache.org/news/2016/05/24/stream-sql.html
- https://www.confluent.io/product/ksql/
- https://samza.apache.org/learn/tutorials/0.14/samza-sql.html
- http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-sql.html
If you enjoyed this post you might also find following interesting.
- https://iwringer.wordpress.com/2015/08/03/patterns-for-streaming-realtime-analytics/
- https://softwareengineeringdaily.com/2016/02/04/stream-processing-vs-complex-event-processing/
- https://medium.com/@srinathperera/second-act-of-realtime-analytics-stream-processors-and-ceps-are-merging-a522649bd742
- https://www.kdnuggets.com/2015/03/sql-query-language-realtime-streaming-analytics.html
Also checkout some of my most read posts at https://iwringer.wordpress.com/ and https://medium.com/@srinathperera. You can follow me at @srinath_perera. You can find my talks at and more details at http://people.apache.org/~hemapani/.
