Kafka Streams — How to build an advanced stateful data stream processing
A practical example of real time account balance calculation using Kafka Streams processor & key-value state store
Given a data stream of transactions in real-time, how do you calculate the latest account balance?
The traditional way could materialize the data stream into a data table and run a calculation logic by a job which polls for new transaction records. The output of this solution depends on how often the job polls on the database. It can at most achieve just a “near real-time” update.
Real time data processing is not possible without data stream programming.
Fast and processing data in real-time is the unparalleled advantage of data streaming. It is critical to many modern systems which are handling huge amounts of data. Delay of just several minutes is not tolerated. Hence, real data stream processing has gradually become the backbone of many enterprise systems.
System logic in the data stream is driven by new transactions and the account balance is calculated and sent to the output data stream in a real time.
A typical and simple data stream handles messages one by one in sequence. It could be data filtering…
